AI Evaluation Engineer

Job Summary

Creating challenging tasks and evaluation criteria for AI coding agents, the part-time AI Evaluation Engineer will work remotely to build realistic developer environments, design solvable tasks, and write tests to verify agent solutions.

Key responsibilities

• Build realistic developer environments with a virtual company context, including codebase and infrastructure
• Design tasks and define success criteria to evaluate AI agent performance in simulated scenarios
• Iterate on tasks and tests based on QA feedback, refining evaluations to ensure fairness and robustness

Required qualifications

• 5+ years of experience in software development
• Proficiency in Python (FastAPI), JavaScript/TypeScript (React), Docker, Postgres, Kafka, and Redis
• Experience in writing functional and integration tests
• English proficiency at a B2 level or higher