AI Evaluation Engineer

Job Summary

To support the advancement of AI systems for public safety agencies, the full-time remote AI Evaluation Engineer will design and maintain automated evaluation pipelines, develop prompt engineering strategies, and analyze model performance to ensure reliability and accuracy in real-world applications.

Key Responsibilities

• Design, build, and maintain automated AI evaluation pipelines for production LLM applications
• Develop prompt engineering strategies and iterate on prompts using quantitative evaluation methods
• Analyze production AI behavior with Python and SQL to identify opportunities for improvement

Required Qualifications

• Must have US citizenship and pass FBI fingerprint and background check in multiple states
• 3+ years of experience in software engineering, machine learning, data science, or a related technical field
• Experience designing evaluation metrics and interpreting AI model performance
• Strong Python development and SQL skills with experience analyzing large datasets
• Experience building or supporting production LLM or Generative AI applications