AI Evaluation Engineer

Job Summary

To support high-quality AI evaluations at scale, the full-time AI Evaluation Infrastructure Engineer will build infrastructure and tooling that enables rapid and reliable assessment of AI models and products, ensuring meaningful results and effective feedback loops.

Key responsibilities

• Build an execution engine for scoring candidate versions against task sets quickly
• Create tooling to validate tasks from production logs before evaluation
• Develop systems for measuring judge-to-human agreement and monitoring drift over time

Required qualifications

• Experience building production platforms or infrastructure, including data pipelines
• Strong statistical literacy, including knowledge of confidence intervals and variance
• Experience evaluating LLM or ML systems, or deep systems engineering experience with an interest in AI evaluation
• Product instinct for internal tools that are user-friendly and effective
• Ability to collaborate across ambiguous product, data, and engineering challenges