Software Development Engineer in Test - Agentic Automation (Remote, CAN)
CrowdStrike, Canada - Remote AB +5
The role
Are you an SDET who wants to define how AI agents are tested, evaluated and trusted in production?
The CrowdStrike Detonations and Threat Analysis Platform Engineering team is hiring a hands-on Software Development Engineer in Test. You'll build and maintain the functional, integration and performance automation that keeps our large-scale, cloud-based cybersecurity platform reliable. You'll also help shape how we test AI. Our core testing frameworks include agentic test generation and analysis, and we use eval-driven development (EDD) to validate the AI agents we build.
What you will do
- Design, develop and maintain automated test suites and frameworks for distributed, cloud-based systems
- Maintain and extend the agentic test generation and analysis tooling built into our major testing frameworks
- Build AI-powered workflows and agent skill files that automate manual test toil, such as test data setup, failure triage and release validation
- Maintain eval suites, golden datasets and scoring pipelines for our AI agents, scoring correctness, tool use, trajectory, faithfulness, safety, latency and cost
- Enforce CI eval gates so that every change to prompts, models, tools or orchestration is measured before release, and feed production failures back into the eval sets
- Detect and mitigate unreliable AI output, including hallucinated assertions, flaky or redundant tests, incorrect failure triage and judge bias
- Design, run and analyze performance, load and stress tests to find bottlenecks and validate scalability across distributed services
- Validate the integration of platform systems, services, data and applications, as well as customer-facing features end-to-end
- Review product, system and agent designs for testability, evaluability, flaws and improvements
- Share knowledge and contribute to the team's quality and evaluation standards
What You’ll Use
- Languages: Java, Node.js, Python, Go
- Data & Messaging: MongoDB, MinIO, RabbitMQ, Kafka
- Cloud & Infrastructure: AWS (EC2, S3, DynamoDB), Docker, Nomad, Terraform, Unix/Linux
- CI/CD & Tooling: Git, Jenkins, GitLab CI/CD, Jira, Bitbucket
- Performance Testing: Locust, k6
- AI/Agentic & Evals: LLM APIs, agent frameworks and LLM eval tooling
What you need
- Degree in Computer Science or equivalent experience (OOP, data structures, algorithms, distributed systems)