AI Research Engineer

Job Summary

To support a growing AI research initiative, the full-time AI Research Engineer will focus on designing and validating evaluation rubrics, developing long-term benchmarks with real user data, and publishing research findings, all while working remotely.

Key responsibilities

• Design and test evaluation rubrics for validity and reliability, calibrating AI judges against human judgment
• Create and maintain long-horizon benchmarks involving real users and tasks, ensuring ongoing accuracy and relevance
• Conduct research on human feedback's impact on AI models and develop methods for practical application in live programs

Required qualifications

• Published research or strong preprints in areas such as evaluation metrics, human-AI interaction, or measurement
• A PhD, a PhD in progress, or an MS with relevant research internships, or equivalent experience in machine learning research
• Strong applied statistics skills, including inter-rater reliability and confidence intervals
• Experience with LLM-as-judge and rubric-based evaluation methodologies
• Proficiency in Python for building research prototypes and evaluation pipelines