Senior/Staff Evaluation Data Analyst (Agentic Search)

Nebius, London, United Kingdom +1

What you will do

Own the quality measurement and improvement cycle end-to-end: identify user needs, define success, build evaluations, diagnose failures, and work with engineers and product managers to improve outcomes..
Design, build, and evolve evaluation systems, including datasets, benchmarks, metrics, and automated pipelines that support development and launch decisions..
Evaluate response quality across multiple dimensions, including relevance, completeness, freshness, factual accuracy, and usefulness to downstream agents..
Investigate quality regressions and metric anomalies, uncover root causes, and identify the improvements with the greatest potential impact..
Help prioritize the product and engineering roadmap, making quality trade-offs explicit and assessing whether changes deliver meaningful improvements for users..

Substantial hands-on experience designing and building evaluation or quality measurement systems for complex technology products, such as search, AI agents, voice technologies, autonomous driving, recommendation systems, advertising, or other similar complex systems..

Nebius, London, United Kingdom +1

Own the quality measurement and improvement cycle end-to-end: identify user needs, define success, build evaluations, diagnose failures, and work with engineers and product managers to improve outcomes..
Design, build, and evolve evaluation systems, including datasets, benchmarks, metrics, and automated pipelines that support development and launch decisions..
Evaluate response quality across multiple dimensions, including relevance, completeness, freshness, factual accuracy, and usefulness to downstream agents..
Investigate quality regressions and metric anomalies, uncover root causes, and identify the improvements with the greatest potential impact..
Help prioritize the product and engineering roadmap, making quality trade-offs explicit and assessing whether changes deliver meaningful improvements for users..

Substantial hands-on experience designing and building evaluation or quality measurement systems for complex technology products, such as search, AI agents, voice technologies, autonomous driving, recommendation systems, advertising, or other similar complex systems..