Production System Engineer

ByteDance, London, England, United Kingdom

The team

The Server Management DevOps team is responsible for the end-to-end lifecycle management of servers across ByteDance’s self-built data centers in the United States and Europe. Our scope covers new hardware introduction, data center delivery, production operations, hardware maintenance, configuration and firmware changes, capacity migration, asset decommissioning, data sanitization, and hardware reuse. The team serves as a central coordination point between multiple functions, including:

Nice to have

Familiarity with technologies such as BIOS/UEFI, BMC, firmware, PCIe, NVMe, NICs, or hardware telemetry.
Proficiency in Python, Go, Bash, or another programming language for production-grade infrastructure automation, including experience designing, building, or maintaining tools and platforms used in large-scale production environments.
Deep knowledge of Linux administration and troubleshooting, preferably Debian or Ubuntu, combined with strong understanding of server architecture and management technologies such as kernels, drivers, BIOS/UEFI, BMC, Redfish, firmware, PCIe, NVMe, NICs, DPUs, hardware telemetry, and failure diagnostics.
Proven hands-on experience introducing and productionizing large-scale GPU infrastructure, including ownership of hardware NPI or fleet onboarding across qualification, system integration, deployment, production validation, operational handoff, and post-launch reliability.
Strong understanding of distributed AI workload behavior and performance analysis, including collective communication, multi-node training, inference serving, GPU scheduling, checkpointing, workload-related bottlenecks, DCGM, NCCL testing, CUDA profiling, and network-fabric telemetry.
Experience building and operating monitoring, telemetry, hardware management, or automated remediation platforms at substantial scale, with measurable improvements in fleet availability, deployment efficiency, incident reduction, operational efficiency, or reliability.
Experience working directly with OEMs, ODMs, component suppliers, or GPU platform vendors throughout qualification, technical escalation, root-cause analysis, and corrective-action processes.
Experience with one or more advanced infrastructure technologies or engineering areas, such as containerisation and orchestration (e.g., Docker, Kubernetes), infrastructure automation frameworks (e.g., Ansible), AI-powered automation, AI agents, Large Language Models, Retrieval-Augmented Generation (RAG), open-source infrastructure projects, technical publications, patents, or relevant industry standards.

Production System Engineer

The team

Production System Engineer

The team

What you will do

What you need

Nice to have

Similar jobs at other companies

Global Production Systems Engineer

Production Systems Engineer

Global Production Platform Engineer

Cloud Production Platform Engineer

Pre Production Engineer

Production Systems Engineer, Sustaining

Company facts

ByteDance