Remember how it used to be to interact with businesses that knew and understood you, that could recommend exactly what you needed, and that with a simple message could get you what you wanted??? Yep... neither do we. That is why at Yalo we are marrying the scale of digital commerce with the personalization and simplicity of conversations to help companies delight their users.
We know that traditional SAAS companies focus on first-world problems... we don't! Having started in Latin America, our roots are in Emerging Markets and therefore we care about bringing amazing experiences to a population that traditionally has been underserved, such as the small shop owner in Brazil who is ordering online for the first time.
If you're looking for a place to make things happen, learn fast, and impact emerging markets in a way that hasn't been done before, look no further. 💫
Come Join us in our mission of improving billions of lives through the power of conversational commerce!
Your mission 🚀
As a Staff Data Scientist, you make our AI agents measurably better, and you prove it. You own how Yalo evaluates agents: evals, trace and error analysis, and the improvement loop that turns what we find in production into fixes to our agent skills, harness, CLI and customers' conversational flows. You work with Product, Engineering, Delivery and Customer Success. Your job is to show what moved the metric, not to guess.
What are the responsibilities for this role?
- Own the evals strategy for our agents: pre-release behavioral testing, production monitoring, and the judges that score conversations.
- Run continuous error analysis on agent traces at scale. Find failure patterns, attribute them to skills, prompts or releases, and rank improvement opportunities by impact.
- Close the loop. Turn findings into fixes and new evals, ship them, and measure the result with sound before-and-after and causal methods.
- Improve our agent skills, harness and CLI using trace evidence. Work through the owning teams' repos and review process.
- Partner with delivery and customer teams to take real customer flows from underperforming to reliably healthy, and make the process repeatable for others.
- Calibrate and validate LLM judges against human review. Measure their stability, report their limits, and set the bar for what counts as evidence at Yalo.
- Write production-quality Python and TypeScript for evals and analysis tooling, with tests and CI, so it can live in a shared codebase.
- Mentor data scientists and engineers on measurement, experimentation and evaluation practice.
- Help shape quarterly and annual planning for agent quality.