Member of Technical Staff
Wafer, San Francisco
The role
Inference performance depends on how efficiently models use the underlying hardware. Techniques across kernels, compilers, runtimes, and serving systems can dramatically improve latency and throughput, but these optimizations are difficult, hardware-specific, and slow to reproduce across new accelerators. And none of it counts until it is running in a customer's production traffic.