Applied Research Engineer, Evaluation

Black Forest Labs, Freiburg (Germany) +1

What you will do

What you focus on will depend on where you're strongest and what the team needs most when you join.

In scope

Building and maintaining the evaluation stack that every model checkpoint has to pass through and prove itself on, from automated benchmarking and LLM-as-judge pipelines, to human evaluation through panels (Prolific and otherwise).
Designing the breadth of evaluation across the taxonomy, from prompt adherence, editing fidelity, aesthetics, text rendering, subject and character consistency, to safety.
Evaluating the evaluation stack itself: checking that human and automated evals agree, that results stay stable over time, and improving them when they don't.
Benchmarking FLUX against the frontier models continuously, while obtaining a meaningful perspective on where we are better or worse than competitors and what such differences really mean.

All jobs

Applied Research Engineer, Evaluation

Black Forest Labs, Freiburg (Germany) +1

What you will do

What you focus on will depend on where you're strongest and what the team needs most when you join.

In scope

Building and maintaining the evaluation stack that every model checkpoint has to pass through and prove itself on, from automated benchmarking and LLM-as-judge pipelines, to human evaluation through panels (Prolific and otherwise).
Designing the breadth of evaluation across the taxonomy, from prompt adherence, editing fidelity, aesthetics, text rendering, subject and character consistency, to safety.
Evaluating the evaluation stack itself: checking that human and automated evals agree, that results stay stable over time, and improving them when they don't.
Benchmarking FLUX against the frontier models continuously, while obtaining a meaningful perspective on where we are better or worse than competitors and what such differences really mean.

What you need

You have made meaningful contributions to the evaluation of generative models already. You understand that a benchmark figure without careful consideration of sample size and judge agreement with humans is at best uninformed optimism and at worst dangerously misleading, and you have been the person that found that inconvenient regression that nobody wanted to talk about.

You likely have

Advanced degree (MS/PhD) in a relevant field or demonstrated equivalent practical experience
Practical experience with evaluating generative models, in a hands-on way. You may have worked with diffusion or flow-matching based image or video models already, but the capability to evaluate them in earnest transfers to other modalities too
Very strong Python and software engineering skills, with pipelines that run reliably and that other people can read if needed
Working knowledge of PyTorch and modern ML infrastructure and practices; you are not a pure Python developer, you know what the modern ML code looks like
Strong grasp of statistics and its shortcomings in practice, and how to work with those; sample sizes, inter-annotator agreement, confidence levels, etc., and a healthy skepticism about what those numbers say about the world
Practical experience with building and operating LLM-as-judge systems, and understanding their limitations
Working knowledge of modern benchmarking practices and of standard datasets and benchmarks in the field already; you have an idea of which to trust and which to disregard
The inclination to look at the actual outputs your system is judging, and to care about what is there beyond an aggregate score

We'd be especially excited if you

Have practical experience with human evaluation, including study design and execution via panels, and analysis of disagreements and annotator performance
Have worked where evaluation meets training (reward models, preference data, RLHF, DPO or similar) and know that whatever you optimize for is what the model learns
Have evaluated agentic or multi-step systems and know how hard it is to attribute success or failure to a single step
Have built tools that people actually want to use internally
Have made public contributions to the field: papers, open source, or otherwise

Applied Research Engineer, Evaluation

What you will do

In scope

Applied Research Engineer, Evaluation

What you will do

In scope

What you need

You likely have

We'd be especially excited if you

Why this role

Everything we do is grounded in four values

About Black Forest Labs

About Black Forest Labs

How we work together

Similar jobs at other companies

Evaluation Engineer, Applied AI

Research Engineer

Member of Technical Staff, Research Engineer

Research Scientist

Research Engineer - Evals

Research Engineer - Model Evaluation & MLOps

Interview questions

Black Forest Labs