Software Engineer - AI Automation
Job Summary
To support the evaluation of AI-powered software development tools, the remote Software Engineer - AI Evaluation & Automation will build and integrate evaluation harnesses, automate benchmark runs, and ensure the reproducibility of results over a six-month contract with a high possibility of extension.
Key Responsibilities
- Build and integrate evaluation harnesses and automation for software development use cases
- Develop versioned, repeatable processes for evaluating AI tools with reproducible run environments
- Analyze benchmarking results, identifying variance and failure patterns to enhance reliability and automation
Required Qualifications
- Strong software engineering background with experience in automation and developer tooling
- Proficiency in at least one general-purpose programming language such as Python, Java, or JavaScript
- Solid knowledge of Git and containerization with Docker
- Experience with APIs, CI/CD pipelines, and typical engineering workflows
- Understanding of AI evaluation methods and potential pitfalls in benchmarking
Complete Job Description
The complete job description is available to members. Premium membership includes:
Full access to 42,189 remote jobs from human-vetted companies, updated daily
Resume Builder - AI-powered tool to craft, enhance, and tailor your resume to a specific job
Twice-monthly live group coaching and the full Remote Career Center
20% member discount on Career Services
Backed by a 30-day money-back guarantee
Company Company Name
Headquarters Headquarters
Founded Founded
Website
BBB URL BBB URL
The company description is available to members. Go Premium for full access to every job and company profile in our remote jobs database, backed by a 30-day money-back guarantee.
