Research Scientist in AI Evaluation

Job Summary

To enhance the quality of frontier AI benchmarks, the full-time Research Scientist in AI Evaluation will identify benchmark weaknesses, develop and validate fixes, and create reusable methods for improvement while working in a hybrid or remote setting.

Key responsibilities

• Red-team frontier benchmarks by identifying and reproducing issues such as reward hacking and task ambiguity
• Develop benchmark repairs by proposing validated fixes and collaborating with benchmark authors for implementation
• Drive research impact by contributing to technical reports, open-source tools, and publications related to benchmark evaluation

Required qualifications

• Strong research or engineering background in machine learning, LLMs, AI agents, and evaluation
• Proficient in Python programming with experience in building and debugging AI agents or evaluation systems
• Ability to identify weaknesses in evaluation methods and rigorously test new approaches
• Experience designing experiments and analyzing model behavior
• Familiarity with LLM/agent benchmarks or automated grading systems