Principal Applied Scientist

The role

We're looking for a Principal Applied Scientist to lead the design and build of an evaluation framework for an enterprise planner-based agent. You will work directly with the client's engineering team to benchmark their agent across planner models, executor LLMs and inference optimization levers. Every proposed change, whether a model upgrade, a harness change or a routing strategy, becomes a controlled, statistically sound experiment with a measurable dollar outcome.

All jobs

Principal Applied Scientist

Robots and Pencils, US Remote +1

Posted pay

Pay not posted

Location: US Remote +1
Work type: Remote
Level: Principal
Skills in the job text: Python
Posted: 2 days ago
Last check: 4 h ago
Job type: Research, Inference & ML systems

Apply on the Robots and Pencils site Make a résumé and a cover letter for this job Sign in to save

The Robots and Pencils site opens in a new tab.

Robots and Pencils

Open tech jobs: 29
With posted pay: 41%
Median posted base: $170K–$228K

About Robots and Pencils: funding, hiring and pay

Robots and Pencils

About Robots and Pencils: funding, hiring and pay

Interview questions

No one has reported an interview at Robots and Pencils yet. Did you interview there? Add the questions.

Check my fit

Compare your résumé with this job. See where you match, what is missing and which lines to change.

Make a résumé and a cover letter for this job

Paste your résumé, or use the résumé on your profile.
Get the résumé as a PDF and the letter as text. We use only the facts in your résumé.
Apply on the Robots and Pencils site.

Open the apply kit

The role

What you need

What You'll Bring

Experience with LLM and agent evaluation methods, such as programmatic state-based verifiers, LLM-as-judge or agent-as-judge, trajectory-level scoring and user simulation
Experience building or evaluating planner-executor or other multi-step, tool-using agent architectures in production, not just prototypes
A strong background in experimental design and statistical inference, including paired designs, non-inferiority testing and power analysis, and the judgment to call a result inconclusive
Experience with multi-objective optimization or cost-aware ML, including Pareto frontier analysis and converting results into projected dollar savings
Familiarity with LLM routing, model cascades or model selection, and an understanding of LLM inference internals and cost levers (prompt and plan caching, context compaction, reasoning budgets, cache-aware model switching)
Experience with observability and tracing for LLM systems, including attributing tokens, cache usage, latency and cost per request
The ability to build production-quality research tooling: Python services, containerized and sandboxed execution, and data pipelines
Prior customer-facing or consulting experience where you scoped and delivered research engagements
The ability to explain technical findings clearly to non-technical audiences and translate them into business impact, and to go a level deeper when someone pushes on your answer instead of staying abstract

Helpful Extras and Unique Skills

Publications or open-source contributions in LLM evaluation, agents or efficient inference
Experience building internal dashboards or web UIs that a client team can adopt as their own
Hands-on use of an LLM gateway or tracing stack, such as LangFuse or LangSmith, to attribute cost and latency at the request level
Experience in a regulated or high-stakes enterprise domain where evaluation rigor carries real consequences
A point of view on cost-quality tradeoffs gained from running them in production rather than reading about them

Our salary range is $200,00 – $300,000

Principal Applied Scientist

The role

Principal Applied Scientist

The role

What you need

What You'll Bring

Helpful Extras and Unique Skills

Why This Role Matters

Evaluation Pipeline & Simulation

Grading, Statistics & Insights

Dashboard, Experiments & Readout

Collaboration & Technical Leadership

Similar jobs at other companies