Inference Optimization Engineer
Hedra, San Francisco
What you will do
- Work directly with research scientists and engineers to make new visual models fast and efficient at inference time.
- Profile model architectures and workloads to understand bottlenecks across compute, memory, communication, and model execution.
- Develop and implement new approaches to improving inference latency, throughput, memory efficiency, and GPU utilization.
- Explore algorithmic optimizations including quantization, sparsity, caching, compilation, attention optimizations, and alternative execution strategies.
- Build or optimize GPU kernels using CUDA, Triton, or similar technologies when existing implementations leave performance on the table.
- Optimize model execution across single-GPU, multi-GPU, and multi-node environments.
- Reason about the interaction between model architecture and hardware, and work with researchers when architectural changes can unlock meaningful performance improvements.
- Investigate communication, memory movement, parallelism, and distributed execution strategies for large visual models.
- Build rigorous benchmarking, profiling, and performance-regression infrastructure to understand performance and evaluate new optimization ideas.