AI Evaluation Engineer
Bask Health 1, United States
Bask Health 1, United States
We're looking for an AI Evaluation Engineer to help ensure that AI-powered features across Bask Health behave accurately, reliably, and consistently before and after they reach production.
Bask Health supports the technology behind direct-to-consumer telehealth businesses, connecting patient experiences, merchant operations, clinical workflows, pharmacy coordination, and commerce capabilities. As we introduce more AI-powered functionality, we need dependable ways to verify that these systems work as intended under realistic conditions.
You'll build and maintain evaluation workflows that test large language models, AI assistants, retrieval systems, and automated features against clearly defined product requirements.
This is a hands-on engineering role that combines AI evaluation, software testing, automation, and production quality assurance. You'll develop test scenarios, evaluation datasets, automated checks, and tools that help engineering teams identify problems before they affect users.
Your work may include testing AI-generated responses, validating structured outputs, evaluating tool-calling behavior, checking retrieval accuracy, and identifying regressions caused by changes to prompts, models, or application code.
You'll collaborate with applied AI engineers, machine learning engineers, backend engineers, product managers, and quality-focused engineering teams to define expected behavior and translate those expectations into repeatable evaluations.
You'll also help establish processes for reviewing AI failures, documenting recurring issues, and determining whether a feature meets its acceptance criteria. You'll work with engineers to reproduce problems and verify that fixes address the underlying issue.
Because Bask operates in healthcare-related environments, you'll pay particular attention to privacy, permissions, unsupported medical claims, and AI behavior that could interfere with approved clinical or pharmacy workflows. Evaluation systems should help verify that AI features remain within their intended operational boundaries.
This position emphasizes practical testing, thoughtful engineering, and continuous improvement. You'll contribute to evaluation standards while independently implementing and maintaining the tools and tests that support AI quality.
We're looking for an engineer who enjoys finding difficult edge cases, understands how AI behavior differs from traditional deterministic software, and wants to help build products that users can depend on.
Bask Health is a software platform that enables entrepreneurs, doctors, and healthcare businesses to launch, manage, and grow direct-to-consumer telehealth businesses.
Our platform brings together patient experiences, clinical workflows, pharmacy operations, and commerce tools in one place.
We're an entrepreneurial, product-focused engineering team that values ownership, technical excellence, and building software that solves real problems. Engineers work closely with product teams and company leadership, take features from idea to production, and have a direct role in shaping the platform.
Engineering at Bask is AI-first and builder-led. We use tools like Cursor and Claude Code throughout development while maintaining high standards for reliability, testing, and software quality.
Ellipsis HealthSan Francisco - Hybrid
$175K–$250K / yr
Posted May 13, 2026
SableSan Francisco
Pay not posted
Posted Sep 15, 2026
BGBxPortugal
€35K–€55K / yr
Posted Oct 16, 2026
Aircall.io, Inc.San Francisco Office
$181K–$250K / yr
Posted Sep 15, 2026
Beghou ConsultingPune +1
Source: the Bask Health 1 Workable job board. We last checked this job 5 h ago. We first saw it 5 h ago. A job on a job board does not always mean that the team interviews people now. We found this job board in a public list. We did not yet compare it with the careers page of the company.
Pay not posted
Posted 24 days ago
Iterative HealthCambridge, Massachusetts, United States, New York, New York +3
$170K–$240K / yr
Posted Sep 18, 2026
No one has reported an interview at Bask Health 1 yet. Did you interview there? Add the questions.