Staff Software Engineer

Job Summary

As the technical lead for inference performance, the full-time Staff Software Engineer will own the end-to-end responsibility for the platform's Token Performance, focusing on optimizing cost per million tokens and increasing per-GPU Goodput while working remotely.

Key responsibilities

• Establish a unified Token Performance metrics framework and build load-testing benchmarks reflecting real production traffic
• Optimize kernel-level inference engines by customizing critical paths and defining boundaries between inference and orchestration layers
• Lead the design of Prefill-Decode disaggregation and analyze the performance impacts of various workloads

Required qualifications

• First-principles understanding of Transformer inference and the KV Cache memory model
• Source-level experience with a mainstream inference engine, with proven ability to identify and fix bottlenecks
• CUDA kernel tuning skills with hands-on experience in memory optimization and profiling
• Bachelor's degree in Computer Science, Electrical/Computer Engineering, or a related field
• Minimum 6 years in systems/Performance Engineering, including 2-3 years on LLM or GPU performance work