AI Test Engineer

• Design, develop, and execute automated test plans and validation scripts for AI-powered applications, LLMs, and autonomous AI agents.
• Formulate testing strategies covering model output accuracy, response relevance, hallucination detection, and prompt injection risks.
• Validate multi-agent system workflows, focusing on task routing, state transition logic, and pipeline error handling.
• Establish benchmarking frameworks to measure AI model accuracy, latency, and performance under stress and load conditions.
• Create mock environments, test datasets, and simulation frameworks to test edge cases and complex agent behaviors.
• Perform continuous regression testing within CI/CD pipelines as AI models are updated or fine-tuned.
• Identify, document, and isolate complex bugs, collaborating with Data Science, Dev, and Product teams to resolve root logic issues.
• Set up real-time automated reporting dashboards to track model quality and system reliability metrics.

**Reporting to: Senior AI engineer**

**Team name: Artificial Intelligence(DSAI), Digital Operation & Engineering**

This role is for an AI Test Engineer who will focus on the end-to-end validation of AI applications, Large Language Models (LLMs), and autonomous AI agents. You will be responsible for creating specialized validation frameworks for conversational systems, multi-agent workflows, and decision-making logic. By combining deep technical skills in AI testing methodologies, automation scripting, and system validation with a passion for quality, you will help ensure our AI agents operate safely, reliably, and accurately before deployment.

• Bachelor’s degree in Computer Science, Software Engineering, or equivalent technical discipline.
• 1 to 3 years of experience in software engineering, testing or test automation, with at least 6 months – 1 year of experience specifically focused on validating AI agents, LLM systems, or conversational agents.

• Design and deployment of AI/LLM evaluation frameworks such as Ragas or Deepeval.
• Advanced test automation scripting using Python with libraries like PyTest, Selenium, or Playwright.
• In-depth understanding of LLM parameters, prompt engineering techniques, agent state machines, and JSON API payloads.
• Software testing methodologies, regression testing automation, and bug lifecycle management in Agile/Scrum environments using JIRA.

• Hands-on experience with CI/CD tools (GitLab CI, GitHub Actions) and containerized systems (Docker, Kubernetes).
• Cloud deployment knowledge on GCP alongside monitoring and logging tools like Grafana.
• Understanding of vector databases (Qdrant, Milvus) and ML tracking platforms (Langfuse, MLflow).
• Testing experience on web or mobile applications featuring integrated AI or conversational interfaces.

Want to know more?

At Ather, we’ve built India’s first intelligent scooter, from scratch. Based on the philosophy of clean design and smart technology, we are transforming the urban commute experience. We are expanding in India, and in the coming years, we look forward to scaling up and launching more products globally. Today, our product offerings include the Ather 450X, 450S, 450 Apex, and Rizta. Beyond the product, we are also setting up an extensive charging network and re-imagining end-to-end ownership experience. To know more, check out our Careers site and our Substack.