Senior HPC Engineer - Fleet Engineering
Lambda, San Francisco Office (Fremont St) +3
The role
If you'd like to build the world's best AI cloud, join us.
What you will do
- Build and operate monitoring and alerting for cluster health — fabric, GPU, power/thermal, and job-level signals — to detect and respond to issues proactively
- Remotely deploy and configure large-scale HPC clusters for AI workloads using automation wherever possible
- Automate cluster lifecycle: operating systems, firmware, drivers, and networking, managed as code (Ansible, Terraform) rather than by hand
- Create runbooks and automated remediations for common cluster failure modes, designed so Support and HPC Support can run them safely
- Troubleshoot and resolve cluster issues across InfiniBand/RoCE, NCCL, GPU-direct, fabric, switching, and power — working closely with on-site deployment teams
- Participate in on-call rotations and lead incident response for cluster-level problems
- Contribute to and maintain Standard Operating Procedures, and feed clear requirements back to other engineering teams on simplification, stability, and operational efficiency