AI Testing Engineer, Amazon Bedrock
NTT DATA, Europe & LATAM, Branch in USA, Inc., LATAM (remote)
The role
Location: 100% remote, seeking candidates located in LATAM.
Contract Length: 1 year, with potential for extension.
Contract Rate: $20-25 per hour in USD. Final rate will be determined based on relevant experience and overall fit for the position.
NTT DATA is seeking an AI Testing Engineer, Amazon Bedrock to define and execute the quality strategy for Generative AI applications and AI agents used in Customer Service and Contact Center scenarios. This role will evaluate AI outputs across deterministic and non-deterministic scenarios using automation, semantic and statistical analysis, human review, and adversarial testing while partnering with AI engineers and functional QA teams.
What you will do
- Define the testing and evaluation strategy for Generative AI applications and AI agents.
- Create and maintain evaluation datasets, golden datasets, and regression suites covering representative business scenarios.
- Design single-turn and multi-turn conversational test cases, including context retention and workflow transitions.
- Evaluate AI outputs for accuracy, relevance, completeness, groundedness, faithfulness, consistency, tone, safety, and compliance.
- Build automated evaluation pipelines and integrate AI-quality checks into CI/CD and release processes.
- Detect regressions introduced by changes to prompts, models, knowledge sources, retrieval strategies, tools, or guardrails.
- Design negative and adversarial test scenarios covering prompt injection, jailbreaks, hallucinations, PII leakage, unsafe responses, and out-of-scope requests.
- Validate fallback behavior, uncertainty handling, tool execution, error handling, and escalation to human agents.
- Perform root-cause analysis for incorrect or undesirable AI outputs and collaborate with AI engineers on remediation.
- Define measurable quality thresholds and release acceptance criteria for AI-agent capabilities.
- Produce quality metrics, evaluation reports, dashboards, and trend analysis for project and product stakeholders.
- Support UAT preparation, defect triage, production monitoring, and continuous improvement of AI behavior.
What you need
Required Experience
- 5+ years of experience in software testing, quality engineering, or test automation with hands-on automation capabilities.
- 2+ years of experience with Amazon Bedrock.
- Practical understanding of Generative AI / LLM behavior and the challenges of testing non-deterministic systems.