Researcher for AI Evaluations

Job Summary

To evaluate frontier AI models on real-world tasks, the full-time Researcher for AI Evaluations will curate a suite of realistic tasks, design grading rubrics, and assess AI performance remotely.

Key Responsibilities

• Create and curate an evaluation suite of real-world tasks for AI capabilities and devise evaluation rubrics
• Regularly evaluate new AI models and products using the task suite, updating tasks and rubrics as needed
• Communicate research findings through public reports, blog posts, and data visualizations to inform team initiatives

Required Qualifications

• Experience with research and data analysis, including light coding skills for result analysis
• Familiarity with AI benchmarks and evaluations, with a grounded understanding of AI capabilities
• Comfort with AI agents and tools, demonstrating experience in working with AI in practical settings
• Strong analytical thinking, ensuring experiments are conducted rigorously and findings are evidence-based
• Excellent written communication skills to convey nuanced observations clearly and precisely