Software Engineering Evaluation Specialist

Job Summary

To support innovative AI development, the part-time Software Engineering Evaluation Specialist will design coding tasks for AI agents, creating Docker environments and verification tests while working remotely on a project-based basis.

Key responsibilities

• Invent realistic developer scenarios and build reproducible Docker environments with specific dependencies
• Write verification tests using pytest and create instructional documentation that mimics developer tickets
• Iterate on tasks based on feedback from QA reviewers and later review other authors' tasks as a QA reviewer

Required qualifications

• 3+ years of production software development experience in one backend stack (Python, Go, Node.js, Java, or Rust)
• Fluency in Python and pytest, with experience in writing tests and debugging
• Experience authoring Dockerfiles and working with Linux and Bash for debugging
• Familiarity with AI coding agents and experience identifying errors in their outputs
• Proficiency in English at a B2+ level