Senior Engineer, Inference Data Plane

Job Summary

To support the expansion of AI Infrastructure, the full-time Senior Engineer, Inference Data Plane will design, develop, and deliver high-scale data plane services for "Inference as a Service," working remotely in a fast-paced environment.

Key responsibilities

• Drive the end-to-end design, development, and delivery of critical data plane components for large generative AI models
• Architect and refine system design proposals for a high-scale AI inference cloud ecosystem, ensuring availability and resiliency
• Implement and optimize distributed inference hosting using advanced techniques like tensor/data parallelism and smart routing

Required qualifications

• Hands-on experience hosting large language or multimodal models using inference engines like vLLM or TensorRT
• Familiarity with distributed inference serving frameworks such as llm-d, NVIDIA Dynamo, or Ray Serve
• Expert-level proficiency in GoLang or Python and familiarity with gRPC
• Proven experience shipping customer-facing software products in a high-scale environment
• Experience integrating and building with open-source software