Senior AI Quality Engineer

Job Summary

To ensure the effectiveness of AI retrieval systems, the full-time remote Senior AI Quality Engineer will build and manage test suites, automate quality checks, and report on system performance while collaborating closely with retrieval engineers.

Key responsibilities

• Build and maintain the golden question suite and negative entitlement test set to ensure system accuracy and compliance
• Automate testing processes to run on schedule and after system changes, publishing results to a quality dashboard
• Train client stewards on the evaluation runbook and assist in interpreting test results and regressions

Required qualifications

• 3+ years of experience in software quality engineering or test automation using Python
• Hands-on experience evaluating search or retrieval-augmented generation (RAG) systems in production
• Experience designing security-focused negative tests related to authorization and data exposure
• Proficiency in AWS, including using Bedrock and OpenSearch APIs
• Strong written communication skills for reporting test outputs to diverse stakeholders