Software Engineering Evaluation Specialist

Job Summary

To support innovative AI projects, the part-time Software Engineering Evaluation Specialist will design coding tasks within Docker environments, challenge AI agents to fix broken software, and ensure tasks are solvable through automated testing and clear documentation, all while working remotely.

Key responsibilities

• Invent realistic developer scenarios and build reproducible Docker environments with pinned dependencies
• Write pytest tests to verify outcomes and create instructional documentation that mimics real developer tasks
• Iterate on task designs based on feedback from expert QA reviewers and later review other authors' tasks

Required qualifications

• 3+ years of production software development experience in one backend stack (Python, Go, Node.js, Java, or Rust)
• Fluency in Python and pytest, with experience in writing tests and using fixtures and parametrize
• Proficiency in Docker authoring, including creating reproducible Dockerfiles and managing dependencies
• Comfort with Linux and Bash for debugging within containers
• Experience with AI coding agents on non-trivial work, with examples of identifying errors