Software Engineer - AI Evaluation
Get new jobs like this one by email. Create a free account for job alerts and to save the jobs you like.
Location: Remote
Compensation: To Be Discussed
Reviewed: Sat, Nov 21, 2026
This job expires in: 30 days
Job Summary
To support a leading AI research organization, the entry-level Software Engineer - AI Evaluation will remotely review Python code and execution logs to diagnose failures in AI coding tasks, with an expected commitment of 35 hours per week.
Key responsibilities
- Read and trace changes in unfamiliar code to evaluate AI coding agents' performance
- Reproduce failures and identify their causes, including issues with implementation, assertions, or environment
- Join daily discussions with research scientists to compare findings and maintain consistent evaluation criteria
Required qualifications
- 2+ years of experience in software writing, reviewing, and debugging, or a completed STEM master's degree or PhD with intensive programming experience
- Hands-on experience with Python and the ability to debug complex codebases
- Familiarity with at least one additional programming language such as JavaScript, Go, C++, or Rust
- Practical experience with Git, Docker, and software build or test environments
- Experience with a test framework like pytest, unittest, Jest, or Mocha
Complete Job Description
The complete job description is available to premium members. Premium membership includes:
Full access to 37,328 remote jobs from human-vetted companies, updated daily
Resume Builder - AI-powered tool to craft, enhance, and tailor your resume to a specific job
Twice-monthly live group coaching and the full Remote Career Center
20% member discount on Career Services
Backed by a 30-day money-back guarantee