Researcher for AI Benchmarks

Job Summary

To develop and publish critiques and reviews of AI benchmarks, the full-time Researcher for AI Benchmarks will analyze new benchmarks, evaluate methodologies, and write public-facing reports in a fully remote environment.

Key Responsibilities

• Review benchmarks by assessing a new benchmark at least every two weeks, evaluating methodologies and implications for AI capabilities
• Publish and maintain reports, writing public-facing updates on new benchmarks and revising them as new versions and models are released
• Dig into data by examining individual tasks within benchmarks in detail, using coding agents while retaining critical oversight

Required Qualifications

• Strong written communication skills with the ability to produce clear, publication-quality drafts
• Critical thinking about research methodologies, with strong opinions on benchmarks and evaluations
• Familiarity with existing AI benchmarks, their methodologies, strengths, and weaknesses
• Comfort with dense data, capable of breaking down benchmarks into their component parts
• Experience writing about AI/ML for a public or research audience is a plus