AI Engineer, Quality

Job Summary

A company is looking for an AI Engineer, Quality (Evals).

Key Responsibilities

• Design and build a unified evaluation platform for agentic systems and audit workflows
• Develop automated pipelines to evaluate new models against critical workflows within hours
• Define and document evaluation standards and advocate for evaluation-driven development

Required Qualifications

• Multiple years of experience shipping production software in complex systems
• Experience with TypeScript, React, Python, and Postgres
• Built and deployed LLM-powered features in production
• Designed observability or tracing infrastructure for AI/ML systems
• Experience with evaluation platforms like LangSmith or Langfuse