AI Evaluation Engineer

Job Summary

To support a project-based initiative, the part-time AI Evaluation Engineer will create realistic developer environments, design evaluation tasks, and write tests to assess AI coding agents remotely.

Key responsibilities

• Build realistic developer environments that simulate a virtual company with a codebase and context
• Design tasks and evaluation criteria, ensuring they are solvable by AI agents
• Write and iterate on tests that verify agent solutions based on QA feedback

Required qualifications

• 5+ years in software development
• Proficiency in Python (FastAPI), JavaScript/TypeScript (React), Docker, Postgres, Kafka, and Redis
• Experience writing functional and integration tests
• English proficiency at B2 level or higher