Technical Staff Member, Coding Agent Evaluation

Job Summary

To advance the evaluation and development of frontier coding agents, the full-time remote Member of Technical Staff, Coding Agent Evaluation will design benchmarks, lead research initiatives, and develop datasets to measure and improve coding model performance.

Key responsibilities

• Design and manage evaluation frameworks for coding agents, including benchmarks and scoring methodologies
• Lead research initiatives to assess and enhance coding model performance across various software engineering tasks
• Analyze model behavior and failure modes to identify weaknesses and implement actionable improvements

Required qualifications

• Strong software engineering background with expertise in Python, C++, or similar programming languages
• 3+ years of experience in software engineering, machine learning, AI research, or related technical fields
• Experience in designing or validating technical assessments and evaluation methodologies
• Familiarity with large language models, coding agents, and model evaluation systems
• Proven ability to build tooling and automate workflows for large-scale experimentation