Senior AI Evaluation Engineer

Job Summary

Joining a dynamic cross-functional development team, the full-time Senior AI Evaluation Engineer will build evaluation harnesses, create golden datasets, and establish production monitoring for AI systems in a remote work environment.

Key responsibilities

• Build the central evaluation harness and templates for consistent testing across teams
• Create golden datasets with input from business and clinical reviewers, ensuring safety and medical accuracy
• Establish production monitoring for drift and regression, auditing evaluations and reporting metrics monthly

Required qualifications

• 4+ years of experience in ML/LLM evaluation, QA engineering for AI systems, or applied research engineering
• Hands-on experience with evaluation frameworks and statistical rigor on small samples
• Ability to set and defend pass thresholds independently of the development team
• Experience with healthcare or safety-critical evaluation is desirable
• Background in red-teaming is a plus