AI Evaluation Engineer

Job Summary

Building high-quality AI evaluation systems, the full-time AI Evaluation Infrastructure Engineer will develop infrastructure and tooling to ensure reliable and efficient evaluations, enabling product teams to make data-driven decisions quickly and confidently in a remote work environment.

Key responsibilities

• Build an execution engine to score candidate versions against task sets rapidly
• Create tooling for task validation and human reviewer calibration
• Develop systems to compare offline evaluation scores with online outcomes and improve evaluation sets

Required qualifications

• Experience in building production platforms or infrastructure, including data pipelines
• Strong statistical literacy, including knowledge of confidence intervals and variance
• Experience evaluating LLM or ML systems or deep systems engineering experience with a focus on AI evaluation
• Product instinct for internal tools that drive team usage and engagement
• Ability to collaborate effectively across product, data, and engineering teams