Inference and ML systems engineer job description
An inference and ML systems engineer makes machine learning models run fast and at low cost. The engineer works on model serving, compilers, GPUs and the software between the model and the hardware.
- Open job postings
- 12,029
- Companies
- 2,715
- Median posted base pay
- $219K
- Experience in the middle posting
- 5 years
- Postings that show pay
- 53%
Inference and ML systems engineer duties and responsibilities
These are the usual duties of the job. One job at one company can have more duties or different duties.
- Build and operate the services that run models for users.
- Measure latency, throughput and cost, and find the slow parts.
- Make models smaller or faster with methods such as quantization and batching.
- Write performance-critical code in C++, Rust or CUDA.
- Plan GPU capacity and schedule the work on the hardware.
- Work with researchers so that new models can run in production.
Skills that employers ask for
The skills in the most inference and ML systems job postings are Python (60%), C++ (28%), PyTorch (26%), AWS (22%) and Kubernetes (22%). A skill name opens the open jobs that ask for it.
- Python60%7,215 jobs
- C++28%3,314 jobs
- PyTorch26%3,096 jobs
- AWS22%2,624 jobs
- Kubernetes22%2,596 jobs
- Linux16%1,976 jobs
- GCP14%1,657 jobs
- Azure14%1,646 jobs
- Go13%1,524 jobs
- C12%1,463 jobs
- Docker11%1,372 jobs
- TensorFlow11%1,345 jobs
The percent is the part of the 12,029 open postings whose text names the skill. We read named languages, tools and platforms. We do not read general skills such as communication. A skill shows here when 3% of the postings or more name it.
Experience, level and work location
The middle posting asks for 5 or more years of experience. The tables show what the open postings state.
Years of experience
| Minimum in the text | Postings | Part |
|---|---|---|
| 0 to 2 years | 639 | 8% |
| 3 to 5 years | 3,597 | 48% |
| 6 to 9 years | 2,137 | 28% |
| 10 years or more | 1,147 | 15% |
From the 7,520 postings that state a minimum.
Level
| Level in the job title | Postings | Part |
|---|---|---|
| Intern | 587 | 5% |
| Entry level | 292 | 2% |
| No level in the title | 5,571 | 46% |
| Senior | 3,441 | 29% |
| Staff | 1,078 | 9% |
| Principal | 663 | 6% |
| Director and above | 397 | 3% |
From all 12,029 postings.
Work location
| Work type | Postings | Part |
|---|---|---|
| On site | 2,588 | 46% |
| Hybrid | 1,558 | 28% |
| Remote | 1,498 | 27% |
From the 5,644 postings that state a work type.
Job titles that employers use
The job titles of this job type that the most companies use in open postings. One job type has many titles.
- Senior Software Engineer126 companies
- Machine Learning Engineer125 companies
- Senior Machine Learning Engineer111 companies
- Software Engineer107 companies
- AI Engineer103 companies
- Data Scientist86 companies
- Member of Technical Staff83 companies
- Senior Data Scientist80 companies
Inference and ML systems engineer salary: posted pay by level
The median posted base pay for inference and ML systems jobs is $219K a year, from 5,814 open jobs at 1,452 companies that state a yearly range in US dollars. The middle half of the jobs is from $183K to $265K. These are pay ranges that employers posted. They are not salaries that people reported, and they do not include stock or a bonus.
| Level | 10th | 25th | Median | 75th | 90th | Jobs with pay |
|---|---|---|---|---|---|---|
| Intern | $66K | $79K | $118K | $154K | $179K | 25 |
| Entry level | $90K | $126K | $151K | $169K | $216K | 78 |
| Level not stated | $144K | $169K | $205K | $254K | $313K | 2,651 |
| Senior | $158K | $188K | $215K | $245K | $271K | 1,765 |
| Staff | $193K | $220K | $249K | $279K | $326K | 686 |
| Principal | $181K | $214K | $256K | $320K | $364K | 384 |
| Director+ | $189K | $231K | $276K | $309K | $353K | 225 |
The companies with the highest median posted base pay for these jobs: Netflix ($608K), Anthropic ($441K), Thinking Machines Lab ($413K), Magic ($413K), Aaru ($376K) and Datadog ($360K).
An outline of a job post that you can copy
Copy this outline and change it for your job. The text in brackets tells you what to write. The duties are the usual duties of the job. The skills, the years and the pay come from the open postings on this page.
Job title: Senior Software Engineer About the job [Two or three sentences: what the team builds, who uses it, and why this job is open.] What you will do - Build and operate the services that run models for users. - Measure latency, throughput and cost, and find the slow parts. - Make models smaller or faster with methods such as quantization and batching. - Write performance-critical code in C++, Rust or CUDA. - Plan GPU capacity and schedule the work on the hardware. - Work with researchers so that new models can run in production. What we look for - [Number] or more years of experience. [The middle posting for this job asks for 5.] - Skills: Python, C++, PyTorch, AWS, Kubernetes, Linux, GCP, Azure. [These are the skills in the most postings. Keep only the skills that the job needs.] - [One or two more requirements that are true for your team.] Pay [Minimum] to [maximum] US dollars a year, and a short description of the bonus, the stock and the benefits. [The middle half of the posted base pay for this job is $183K to $265K a year.] Work location [On site, hybrid or remote. The city and the state.] How to apply [The link, and what happens after a person applies.]
Pay in the job post: the rule in your state
14 US states and the District of Columbia require a pay range in a job post: Colorado, California, Washington, New York, Hawaii, District of Columbia, Maryland, Illinois, Minnesota, New Jersey, Vermont, Massachusetts, Virginia, Maine and Connecticut. Many of these laws also cover a remote job that a person in the state can do. See the pay transparency law of each place, with the official source.
Do you hire for this job?
Put your job in front of people who look for inference and ML systems jobs with the pay shown.
Post this job freeDo you look for this job?
12,029 inference and ML systems jobs are open now. 53% of them show the pay.
See the open inference and ML systems jobsNewest inference and ML systems jobs
Read real postings to see how employers describe this job. Each job is from the job board of the company.
- Agentic AI EngineerDefense Unicorns · No posted pay
- Senior Data Engineer 2026 - USAimpoint Digital · No posted pay
- Lead Data Engineer 2026- US, COL, UKAimpoint Digital · No posted pay
- Machine Learning InternAftershoot · No posted pay
- Principal Research Engineer/ScientistServiceNow · $276K–$483K / year
- Software Engineer, SimulationAeroVect · $110K – $170K • Offers Equity
Do you want to apply? See the inference and ML systems engineer résumé example.
More about inference and ML systems jobs
Inference and ML systems engineer
Other job descriptions
How we made this page
The numbers come from open job postings. We read each posting from the job board of the company and check it every day. A program reads the skills, the years of experience and the level from the text and the title of each posting. One job can be in more than one job type.
A person wrote the short description of the job and the list of duties. They are general: one job at one company can be different. The pay numbers are posted base pay ranges in US dollars for each year. They do not include stock or a bonus, and they are not salaries that people reported.