Remote Jobs Sign In

Software Engineer for AI Evaluation

Get new jobs like this one by email. Create a free account for job alerts and to save the jobs you like.

Location: Remote
Compensation: Hourly
Reviewed: Thu, Nov 19, 2026
This job expires in: 30 days

Job Summary

To support the evaluation and automation of AI-powered software development tools, the full-time Software Engineer for AI Evaluation will build evaluation harnesses, automate benchmark runs, and ensure the reproducibility of results while working remotely within PST and CST time zones.

Key Responsibilities
  • Build and integrate evaluation harnesses and automation for software development use cases
  • Establish versioned, repeatable processes to evaluate AI tools and models with reproducible environments
  • Analyze results from benchmark runs to improve workflows and ensure consistency and accuracy of evaluations
Required Qualifications
  • 5 years of experience in building automation, developer tooling, or test and validation systems
  • Proficiency in Python and familiarity with at least one of Java, JavaScript, or a similar language
  • Solid knowledge of Git and experience with containerization using Docker
  • Experience with APIs, CI/CD pipelines, and typical engineering workflows
  • Familiarity with AI, LLM, or agent evaluation concepts and their potential pitfalls

Complete Job Description

The complete job description is available to premium members. Premium membership includes:

Full access to 38,909 remote jobs from human-vetted companies, updated daily

Resume Builder - AI-powered tool to craft, enhance, and tailor your resume to a specific job

Twice-monthly live group coaching and the full Remote Career Center

20% member discount on Career Services

Backed by a 30-day money-back guarantee