QA Engineer, AI Quality

Job Summary

To support a growing AI-driven platform, the part-time contract QA Engineer, AI Quality will inherit and stabilize the Playwright end-to-end testing suite, build evaluation systems for AI output quality, and ensure actionable reporting in a remote environment with consistent EST overlap.

Key responsibilities

• Manage Playwright automation by auditing and extending the existing end-to-end testing suite
• Develop and implement evaluation rubrics for AI-generated card outputs, ensuring quality and consistency
• Provide concise reporting and actionable insights based on evaluation findings for the AI engineering team

Required qualifications

• 3 to 5 years of experience in QA engineering with ownership of an automation framework
• Strong proficiency in Playwright and experience with visual or multimodal AI output evaluation
• Experience building and managing LLM evaluation pipelines, including rubric design and regression tracking
• Full stack literacy to test modern web applications, specifically Next.js/React and Python/FastAPI
• Understanding of RAG and AI orchestration systems, with the ability to create meaningful tests and evaluations