LLM Evaluation Expert

Job Summary

A company is looking for an LLM Evaluation Expert (Coding).

Key Responsibilities:

• Evaluate LLM-generated code across various programming languages and paradigms
• Select optimal coding solutions from LLM outputs, aligning with client needs and industry standards
• Develop benchmarks for "what good looks like" in AI-generated code through writing demonstrations

Required Qualifications:

• Advanced degree in Computer Science with 5+ years of extensive experience in software development
• Proven ability to evaluate code in multiple programming languages including Python, PHP, Java, and others
• Experience in understanding user requests and evaluating model responses based on correctness and logic
• Background in software engineering with experience in AI systems related to code generation
• Familiarity with machine learning concepts, particularly in natural language processing