AI Agent Evaluation Analyst

Job Summary

A company is looking for an AI Agent Evaluation Analyst (Freelance).

Key Responsibilities

• Reviewing evaluation tasks and scenarios for logic, completeness, and realism
• Identifying inconsistencies, missing assumptions, or unclear decision points
• Helping define clear expected behaviors (gold standards) for AI agents

Required Qualifications

• Excellent analytical thinking about complex systems and logical implications
• Familiarity with structured data formats, such as JSON/YAML
• Ability to assess scenarios holistically and identify missing or unrealistic elements
• Experience with policy evaluation, logic puzzles, or structured scenario design
• Exposure to LLMs, prompt engineering, or AI-generated content is a plus