Director of AI Evaluation

Job Summary

Leading a multidisciplinary team, the full-time salaried Director of AI Evaluation will define and uphold quality standards across AI initiatives, manage data scientists and analysts, and ensure robust evaluation methodologies for both internally built and vendor-provided AI systems while working remotely.

Key responsibilities

• Determines and enforces quality standards for high-value AI initiatives from design through production
• Provides technical guidance for validation studies and monitors AI system performance against established metrics
• Develops and retains a strong technical team while translating ambiguous failure modes into measurable evaluation designs

Required qualifications

• Bachelor's Degree in a related field
• Minimum of 8 years of related work experience, with at least 3 years in a managerial role
• Strong foundation in experimental design and causal inference
• Hands-on experience in evaluating AI systems in production settings
• Proficiency in Python and SQL, with familiarity in modern ML tooling and cloud environments