RAG and Evaluation Engineer

Job Summary

To support a critical legacy modernization project, the remote RAG and Evaluation Engineer will manage knowledge surface ingestion pipelines, ensure retrieval quality, and own evaluation benchmarks while collaborating with a senior engineering team focused on applying AI to enhance a decades-old codebase.

Key responsibilities

• Own the knowledge surface ingestion pipelines for source code, structured metadata, and technical documentation
• Ensure retrieval quality through chunking, embeddings, hybrid retrieval, and reranking
• Manage evaluation benchmarks for translation accuracy and overall agent quality

Required qualifications

• Bachelor's degree in Computer Science, Engineering, Information Science, or a related field, plus 4 years of professional software engineering experience
• Experience shipping a production RAG system with measurable quality metrics
• Production experience with retrieval pipelines, chunking, embedding, and hybrid retrieval
• Strong applied evaluation skills, including benchmark design and regression detection
• Proficiency in TypeScript or Python and experience in a remote work environment