Senior Site Reliability Engineer - AI Compute

Job Description The Data Infrastructure SRE team is responsible for the reliability, scalability, and efficiency of the compute platform that powers our products and data workloads. We operate large-scale Kubernetes environments and AI infrastructure across heterogeneous CPU and GPU resources, including cluster control planes, worker nodes, workload scheduling, container runtimes, and resource management systems. Our focus is to keep shared compute services available, predictable, and cost-efficient while enabling product and platform teams to run critical workloads safely at scale.

This role includes participation in a rotational on-call schedule to provide continuous coverage for critical compute infrastructure. You will respond to, troubleshoot, and resolve production incidents involving Kubernetes clusters, CPU and GPU nodes, containers, scheduling, and resource availability. Our team works across multiple time zones and follows rigorous change management and post-incident review practices to maintain platform stability.

As a Site Reliability Engineer focused on compute and AI Infra, you will operate and improve large-scale Kubernetes platforms, manage cluster and node lifecycle changes, diagnose workload and scheduling failures, and automate repetitive tasks. You will work with senior engineers and partner teams to improve the reliability, performance, and efficiency of the compute platform.

• Kubernetes and AI Infra reliability: Own the reliability of large-scale Kubernetes and GPU clusters. Lead incident response, root-cause analysis, and corrective actions across control planes, nodes, container runtimes, scheduling, and host-level dependencies.
• Cluster and node lifecycle management: Operate CPU and GPU clusters through provisioning, upgrades, scaling, configuration changes, repair, replacement, and decommissioning. Build safe rollout, rollback, and validation mechanisms for cluster-level changes.
• Heterogeneous resource management and scheduling: Improve placement, utilization, isolation, and availability for online, batch, data, and AI/ML workloads across CPU and GPU resources. Optimize quotas, priorities, preemption, affinity, autoscaling, and accelerator-aware scheduling.
• Capacity, performance, and cost: Own compute capacity planning, resource forecasting, and performance tuning across clusters and failure domains. Balance workload demand, reliability, utilization, and infrastructure cost.
• SLOs, observability, and operational readiness: Define SLOs and error budgets for compute and AI Infra services. Improve monitoring, alerting, runbooks, production-readiness reviews, and failure recovery for clusters and workloads.
• Automation and technical leadership: Build maintainable tooling and automated remediation to reduce operational toil and improve deployment safety. Partner with workload owners and infrastructure teams on platform standards, design reviews, and operational practices.

• Bachelor’s degree in Computer Science, a related technical field, or equivalent practical experience.
• 3+ years of experience in Site Reliability Engineering, Production Engineering, Systems Engineering, or a similar infrastructure role.
• Strong hands-on experience operating Kubernetes in production, including troubleshooting clusters, nodes, pods, controllers, and workload scheduling.
• Experience managing compute resources across CPU and GPU nodes, including capacity, allocation, isolation, utilization, and failure recovery.
• Strong knowledge of Linux systems, container runtimes, cgroups, namespaces, networking fundamentals, and host-level performance troubleshooting.
• Proficiency in Go, Python, Bash, or another programming language used for automation and infrastructure tooling.

• Experience operating large, multi-cluster or multi-region Kubernetes fleets, including GPU clusters and AI Infrastructure at scale.
• Deep knowledge of Kubernetes architecture and internals, including the API server, etcd, scheduler, controller manager, kubelet, admission control, and custom controllers or operators.
• Experience with heterogeneous resource scheduling and management, including CPU/GPU requests and limits, quotas, priorities, preemption, affinity, taints and tolerations, autoscaling, bin packing, workload isolation, and accelerator-aware placement.
• Experience supporting distributed AI/ML training or inference workloads and diagnosing GPU availability, utilization, topology, device-plugin, or scheduling issues.
• Experience with container runtimes, infrastructure-as-code, configuration management, and Kubernetes deployment tooling such as Terraform, Ansible, Helm, or GitOps workflows.
• Experience designing AI Infra reliability mechanisms, including SLOs, capacity models, failure-domain planning, observability, automated remediation, and workload recovery.
• A systematic approach to complex troubleshooting, strong written and verbal communication, and clear ownership during high-impact incidents.