QA Automation Engineer
Get new jobs like this one by email. Create a free account for job alerts and to save the jobs you like.
Job Summary
To support the growth of AI-driven products, the full-time QA Automation Engineer will build and maintain automated testing for AI voice and chat agents, validate LLM output quality, and ensure release quality in a fully remote environment.
Key responsibilities
- Build and maintain automated testing for AI voice and chat agents, covering various conversational scenarios
- Test and validate LLM output quality and retrieval relevance using evaluation harnesses and structured testing methods
- Own release quality by implementing regression strategies and ensuring early detection of issues
Required qualifications
- 5+ years as a QA Automation Engineer with a focus on testing AI systems
- Proven experience in building testing frameworks and CI/CD-integrated automation from scratch
- Hands-on experience testing conversational AI projects, including voice and chat agents
- Strong understanding of LLMs, prompt engineering, and retrieval-augmented generation (RAG)
- Proficiency in UI/E2E and API test automation, along with experience in database and performance testing
Complete Job Description
The complete job description is available to premium members. Premium membership includes:
Full access to 36,860 remote jobs from human-vetted companies, updated daily
Resume Builder - AI-powered tool to craft, enhance, and tailor your resume to a specific job
Twice-monthly live group coaching and the full Remote Career Center
20% member discount on Career Services
Backed by a 30-day money-back guarantee