Remote Jobs Sign In

Software Engineer - AI Evaluation

Get new jobs like this one by email. Create a free account for job alerts and to save the jobs you like.

Location: Remote
Compensation: To Be Discussed
Reviewed: Sat, Nov 21, 2026
This job expires in: 30 days

Job Summary

To support a leading AI research organization, the entry-level Software Engineer - AI Evaluation will remotely review Python code and execution logs to diagnose failures in AI coding tasks, with an expected commitment of 35 hours per week.

Key responsibilities
  • Read and trace changes in unfamiliar code to evaluate AI coding agents' performance
  • Reproduce failures and identify their causes, including issues with implementation, assertions, or environment
  • Join daily discussions with research scientists to compare findings and maintain consistent evaluation criteria
Required qualifications
  • 2+ years of experience in software writing, reviewing, and debugging, or a completed STEM master's degree or PhD with intensive programming experience
  • Hands-on experience with Python and the ability to debug complex codebases
  • Familiarity with at least one additional programming language such as JavaScript, Go, C++, or Rust
  • Practical experience with Git, Docker, and software build or test environments
  • Experience with a test framework like pytest, unittest, Jest, or Mocha

Complete Job Description

The complete job description is available to premium members. Premium membership includes:

Full access to 37,328 remote jobs from human-vetted companies, updated daily

Resume Builder - AI-powered tool to craft, enhance, and tailor your resume to a specific job

Twice-monthly live group coaching and the full Remote Career Center

20% member discount on Career Services

Backed by a 30-day money-back guarantee