Software Engineer for AI Evaluation
Get new jobs like this one by email. Create a free account for job alerts and to save the jobs you like.
Job Summary
To support the evaluation and automation of AI-powered software development tools, the full-time Software Engineer for AI Evaluation will build evaluation harnesses, automate benchmark runs, and ensure the reproducibility of results while working remotely within PST and CST time zones.
Key Responsibilities
- Build and integrate evaluation harnesses and automation for software development use cases
- Establish versioned, repeatable processes to evaluate AI tools and models with reproducible environments
- Analyze results from benchmark runs to improve workflows and ensure consistency and accuracy of evaluations
Required Qualifications
- 5 years of experience in building automation, developer tooling, or test and validation systems
- Proficiency in Python and familiarity with at least one of Java, JavaScript, or a similar language
- Solid knowledge of Git and experience with containerization using Docker
- Experience with APIs, CI/CD pipelines, and typical engineering workflows
- Familiarity with AI, LLM, or agent evaluation concepts and their potential pitfalls
Complete Job Description
The complete job description is available to premium members. Premium membership includes:
Full access to 38,909 remote jobs from human-vetted companies, updated daily
Resume Builder - AI-powered tool to craft, enhance, and tailor your resume to a specific job
Twice-monthly live group coaching and the full Remote Career Center
20% member discount on Career Services
Backed by a 30-day money-back guarantee