Responsibilities
Your future team
Studio is where Atlassian customers build, run and manage AI agents under the Rovo brand. Rovo Agents is a model-agnostic cloud agent platform. Teams give an agent instructions, knowledge, tools and skills, then put it to work in Jira and Confluence, inside automation rules, or as an autonomous agent that works on its own.
Because customers build these agents themselves, quality can't be checked once before release. Every agent has its own instructions, data and tools, and any of them can be misconfigured or connected to a tool that fails. We ask three questions of every agent: did it understand and follow the task, did it use its tools correctly and report honestly what it did, and did the result actually help?
The team is software and machine learning engineers based in Sydney and Melbourne, with remote options. We work closely with product, data science and Atlassian's central AI teams. This is applied ML on real agentic systems in production.
What you'll do
As a Senior Machine Learning Engineer, you will design, build, ship and operate the evaluation and optimization systems for Rovo agents.
- Build offline evaluation suites and benchmarks that reflect realistic agent tasks, using real usage patterns, synthetic scenarios and simulated tools. Run them as regression tests before changes ship.
- Build online quality metrics on production traffic. This covers sampling strategy, cheap behavioral signals such as tool errors, retry loops, failed handoffs and abandoned sessions, and model-based judging of selected agent trajectories.
- Calibrate judges against human ratings and independent outcome signals, and evolve evals so they don't saturate as agents improve.
- Use evaluation results and production signals to improve agent quality across prompts, context selection, tool use, retrieval and agent routing, including finding the right agent for a request.
- Improve quality while reducing cost and latency across production LLM systems.
- Own projects from problem definition through experimentation, deployment and ongoing measurement. Contribute to design and code reviews, coach engineers, and help software engineers on the team write good evals.