Inference Software Engineer
Etched, Cupertino, CA +1
What you will do
- Contribute to the architecture and design of the Sohu host software stack
- Implement high-performance, modular code across the complete Etched software stack, consisting of a mix of Rust, C++ and Python.
- Interface with firmware and drivers teams delivering highest-performance HW/SW stack.
- Work with AI model researchers and product-facing teams building out the Etched serving front-end.
What you need
- Experience with C++ and Python
- Familiarity with transformer model architectures and inference serving stacks (vLLM, SGLang, etc.) or experience working in distributed inference/training environments
- Experience working cross-functionally in large software and hardware organizations
Representative projects
- Build scheduling logic for handling continuous batching and real time inference
- Implement inference-time acceleration techniques such as speculative decoding, tree search, KV cache sharing, etc.