AI Agent Evaluation Analyst

Job Summary

A company is looking for an AI Agent Evaluation Analyst (Freelance).

Key Responsibilities

• Review evaluation tasks and scenarios for logic, completeness, and realism
• Identify inconsistencies, missing assumptions, or unclear decision points
• Help define clear expected behaviors (gold standards) for AI agents

Required Qualifications

• Excellent analytical thinking regarding complex systems and scenarios
• Familiarity with structured data formats, such as JSON/YAML
• Ability to assess scenarios holistically for missing or unrealistic elements
• Experience with policy evaluation, logic puzzles, or structured scenario design
• Exposure to LLMs, prompt engineering, or AI-generated content