LLM Evaluation Expert

Job Summary

A company is looking for an LLM Evaluation Expert (Coding) - Part Time.

Key Responsibilities:

• Evaluate LLM-generated code across various programming languages and paradigms
• Select optimal coding solutions from LLM outputs, aligning with client needs and industry standards
• Develop benchmarks for "what good looks like" in AI-generated code through writing demonstrations

Required Qualifications:

• Advanced degree in Computer Science with 5+ years of extensive experience in software development across multiple languages and paradigms
• Proven ability to evaluate code in several programming languages such as Python, PHP, Java, Ruby, JavaScript, TypeScript, C++, Go, Cypher, SQL
• Experience in understanding user requests and evaluating model responses based on correctness, logic, coherence, and applicability
• Background in software engineering with experience in AI systems, particularly in code generation
• Familiarity with machine learning concepts related to natural language processing is advantageous