- Design, run and maintain end-to-end business scenarios that exercise multi-step agentic workflows,
including agent routing, tool/function calls, and multi-turn conversations.
- Evaluate agent outputs for correctness, relevance, groundedness, consistency, tone and
completeness, using both human review and automated methods.
- Build and maintain eval datasets (golden sets, edge cases, adversarial prompts) and automated
regression suites that run as agents, prompts, models and tools change.
- Identify cross-agent, orchestration and integration issues: wrong agent selection, broken handoffs,
tool-call failures, context loss, looping or runaway behavior.
- Test for hallucination, prompt injection, jailbreaks, data leakage and unsafe or out-of-scope responses.
- Detect regressions caused by model upgrades, prompt changes or retrieval/knowledge changes, and
quantify them with metrics.
- Triage and root-cause issues with Engineering and Product, separating prompt, model, retrieval, tool
and orchestration causes.
- Re-test fixes, validate the end-user experience, and keep evals current as the product evolves.
- Document test strategy, eval results and quality trends clearly for global stakeholders.