AI Quality Analyst

Job Summary

Partnering with clients during implementation, the full-time AI Quality Analyst will assess the quality and safety of AI-generated responses, translating client requirements into evaluation frameworks while working remotely.

Key responsibilities

• Review and score AI responses using standard rubrics and validate against sources to identify risks
• Translate client requirements into evaluation criteria and test coverage, acting as a quality liaison between clients and technical teams
• Document findings and collaborate with peers to ensure consistent scoring and address quality issues proactively

Required qualifications

• Experience in AI evaluation, data annotation, or quality assurance in a regulated environment
• Familiarity with large language models and their common failure modes
• Strong analytical judgment with the ability to apply detailed guidelines consistently
• Experience in client-facing roles such as implementation or consulting in a SaaS environment
• Ability to translate business needs into technical evaluation criteria