Senior Data Scientist Evaluation

Job Summary

To enhance the evaluation processes for AI models, the remote Senior Data Scientist Evaluation will design ground truth metrics, build repeatable evaluation loops, and collaborate with cross-functional teams to drive system improvements in a full-time capacity.

Key responsibilities

• Design AI evaluations by defining ground truth, metrics, and scoring methods for models and agents
• Build repeatable evaluation loops to track quality over time and catch regressions before release
• Drive iteration by translating evaluation results into actionable recommendations for system improvements

Required qualifications

• 6-10 years of experience in quantitative data science or machine learning, with a focus on measurement or evaluation
• Strong statistical and machine learning foundation, including experience with experimental design and validation
• Proficiency in Python and SQL, particularly in evaluating models in production environments
• Demonstrated ability to define quality metrics and ground truth for ambiguous outputs
• Excellent communication skills for effective cross-functional collaboration