Evals Lead, AI Observability

Job Summary

To support project teams in evaluating LLM-based applications, the remote contract Evals Lead, AI Observability will provide evaluation expertise, develop enablement content, and offer consultancy on evaluation design while ensuring accountability remains with the individual teams.

Key responsibilities

• Develop enablement and adoption content to guide teams in setting up evaluations, including best practices and methodologies
• Create reusable evaluation patterns and templates to address common evaluation challenges across projects
• Consult with project teams on evaluation design, reviewing datasets and measurement strategies while advising on soundness and thresholds

Required qualifications

• Direct experience designing and running LLM evaluations, including dataset construction and measurement decisions
• Understanding of LLM-as-judge practices and its calibration against human judgment
• Knowledge of sampling strategies, annotation guidelines, and managing dataset quality over time
• Strong grasp of statistical measurement concepts, including sample size and variance
• Familiarity with LLM application architecture and evaluation tooling such as Langfuse or similar platforms