Model Serving Engineer

Job Summary

Seeking a full-time Model Serving Engineer to design, build, and operate high-performance inference platforms for large machine learning models in production, working remotely and focusing on systems engineering aspects such as request routing, batching, and GPU utilization.

Key Responsibilities

• Design and operate model serving platforms for diverse workloads, including LLMs and recommendation systems
• Optimize inference performance through techniques like continuous batching and request multiplexing
• Implement multi-tenant routing and capacity management systems to balance latency and cost

Required Qualifications

• Bachelor's or Master's degree in Computer Science or a related field
• Six or more years of experience in distributed systems or ML platform engineering
• Strong proficiency in Python and a systems language such as Go, Rust, or C++
• Hands-on experience with large model inference frameworks like vLLM or TensorRT-LLM
• Familiarity with Kubernetes and modern cloud platforms