AI Evaluation Analyst

Job Summary

A company is looking for an AI Agent Evaluation Analyst (Freelance).

Key Responsibilities

• Review evaluation tasks and scenarios for logic, completeness, and realism
• Identify inconsistencies, missing assumptions, or unclear decision points
• Help define clear expected behaviors (gold standards) for AI agents

Required Qualifications

• Excellent analytical thinking with the ability to reason about complex systems
• Familiarity with structured data formats, such as JSON/YAML
• Experience with policy evaluation, logic puzzles, or structured scenario design
• Background in consulting, academia, or research
• Exposure to LLMs, prompt engineering, or AI-generated content