Software Engineering Evaluation Specialist

Job Summary

To support innovative AI projects, the part-time Software Engineering Evaluation Specialist will design coding tasks for AI agents, creating self-contained Docker environments and ensuring tasks are solvable while working remotely.

Key responsibilities

• Invent realistic developer scenarios and build reproducible Docker environments with pinned dependencies
• Write comprehensive pytest tests and instructional documentation that mimic real developer tasks
• Iterate on task difficulty based on feedback from expert QA reviewers and later review other authors' tasks

Required qualifications

• 3+ years of production software development experience in a backend stack such as Python, Go, Node.js, Java, or Rust
• Fluency in Python and pytest, with experience in writing tests that are deterministic and non-flaky
• Proficiency in Docker authoring and creating reproducible Dockerfiles
• Comfort with Linux and Bash for debugging inside containers
• Experience with AI coding agents on non-trivial work, demonstrating the ability to identify errors