Inference Engineering, Co-op

Inferact, San Francisco

The role

Overview

Inferact's mission is to grow vLLM as the world's AI inference engine and accelerate AI progress by making inference cheaper and faster. Founded by the creators and core maintainers of vLLM, we sit at the intersection of models and hardware, a position that took years to build.

Potential Focus Areas

Your co-op will have a primary home in one of the following tracks, with opportunities to contribute across others:

Inference Runtime: Bring new model architectures and inference techniques to life in vLLM. Implement ideas from research papers; support mixture-of-experts, multimodal, diffusion, and agentic workloads; and improve scheduling, continuous batching, KV-cache memory management, prefix caching, and hybrid model serving.
Performance & Scale: Build the distributed serving data plane that lets vLLM run across many GPUs and nodes. Work on tensor, expert, or context parallelism; prefill/decode separation and KV-cache transport; fault tolerance and multi-tenancy; and high-performance communication using NCCL, DeepEP, NVSHMEM, RDMA, or InfiniBand.
Kernel Engineering: Raise the hardware performance ceiling by writing and optimizing attention, GEMM, sampling, KV-cache, fused, and quantization kernels. Use CUDA, Triton, TileLang, CUTLASS/CUTE, or related tools; reason about memory hierarchy, occupancy, and tensor cores; and prove speedups through profiling, correctness tests, and reproducible benchmarks.
AMD GPU Performance: Help make vLLM first-class on AMD accelerators. Work across ROCm, HIP, Triton, CK, AITER, kernels, runtime paths, quantization, compiler integration, and performance-regression infrastructure while learning how AMD-specific execution, memory, and toolchain constraints shape inference.
TPU Performance: Help make vLLM fast and correct on Google TPUs. Build backend, runtime, and compiler integrations with JAX, XLA, Pallas, MLIR, and related tooling; inspect compiler artifacts; work on lowering, fusion, and code generation; and benchmark production-relevant serving across correctness, latency, and throughput.
Cloud Orchestration: Build the operational platform that makes large-scale inference deployable and reliable. Work on Kubernetes and custom operators, topology-aware GPU scheduling, zero-downtime vLLM rollouts, token-aware routing, observability, infrastructure-as-code, automated recovery, and bring-your-own-cloud or multi-cloud fleet management.

You are not expected to arrive with experience in every track. We care most about unusual depth or learning velocity in one area, strong fundamentals, and evidence that you can turn a difficult problem into working, measurable software.

Inference Engineering, Co-op

The role

Inference Engineering, Co-op

The role

What you need

Nice to have

Preferred qualifications

Bonus points if you have

Potential Focus Areas

Meet Inferact at Waterloo

Logistics

Similar jobs at other companies

LLM Inference Systems Engineer

Software Engineer, Inference Runtime

Machine Learning Intern/Co-op (Winter 2027)

Working Student - LLM Inference Performance

Research Intern, Inference (Summer 2027)

Machine Learning Researcher

Inferact