Evaluations - Member of Technical Staff

Simile, San Francisco +1

The role

As a Member of Technical Staff, Model Evaluations at Simile, you will help define how simulations of human behavior should be tested and when their predictions are useful enough to guide real decisions. You will develop measurement methods, challenge existing assumptions, and turn evidence into priorities for modeling and product. This is an opportunity to help establish evaluation practices for an emerging field while building systems that people use.

What you will do

You will own a defined evaluation problem or workstream, with scope and independence matched to your experience. Depending on the workstream, you may build on existing datasets and benchmarks or develop new studies, data resources, and evaluation infrastructure. Your work may include:

Build the measurement layer for behavioral simulation: Design evals, metrics, rubrics, datasets, dashboards, and workflows that measure whether Simile’s models are accurately predicting human behavior across customer use cases, populations, question types, and decision contexts.
Partner with modeling to improve models: Evaluate new model versions, diagnose regressions, identify priority areas for model-improvement cycles, and maintain stable eval suites that represent capabilities customers actually care about.
Contribute to product and applied evals: Build evals for qualitative responses, retrieval, survey generation, AI-generated research reports, customer-facing outputs, and other product surfaces where model quality directly shapes customer trust. Turn subjective quality concerns into concrete rubrics, labeled data, automated graders, release criteria, and model-improvement signals.
Make ground truth and uncertainty legible: Develop rigorous ways to compare simulated responses against human data, customer studies, Simile-collected ground truth, and behavioral datasets. Help the company reason about sampling error, uncertainty, calibration, margin of error, representativeness, and what “ground truth” means when human behavior is inherently noisy.
Automate evaluation workflows: Use modern agentic coding tools to rapidly build internal tools, inspect model outputs, create labeling workflows, validate evals, and turn fuzzy evaluation questions into working systems. We value people who can compress long, ambiguous projects into fast, useful prototypes without losing sight of rigor or reliability.
Expand what we can evaluate: Develop repeatable benchmarks for behavioral predictions across transaction and product-interaction data, intervention effects, individual trajectories, and emerging multi-agent settings.

What you need

Evaluation and Measurement Judgment: You can show how you turned an ambiguous quality question into a validated measurement that informed a decision. You investigate surprising results by inspecting individual examples, checking data and assumptions, and testing whether the evaluation itself is misleading. You compare against meaningful baselines, explain what your measurement misses, and revise your methods or conclusions when the evidence warrants it. Examples can come from research, industry, or independent projects.
LLM and Model Fluency: You can work with LLM outputs, understand modeling team needs, and reason about whether a model change improved the behavior we care about. You can learn unfamiliar model concepts quickly and collaborate effectively on evaluation and model improvement. Prior specialization in LLM training or post-training is not required.
Statistical Judgment: You are comfortable reasoning about noisy data, uncertainty, sampling, distributions, calibration, confidence intervals, measurement validity, bias, variance, and the difference between an observed result and the underlying population quantity it estimates.
Technical and Agentic Execution: You can build internal tools, scripts, dashboards, labeling workflows, analyses, or automated eval pipelines quickly. You are comfortable working with data and automation tools such as Python, SQL, R, notebooks, LLM APIs, and agentic coding tools such as Codex, Claude Code, Cursor, or equivalent systems. You know how to move quickly while still validating outputs, catching errors, and planning for the long term.
Hands-On Ownership and Collaboration: You drive workstreams while doing the work yourself: building prototypes, inspecting data, debugging workflows, and improving methods. When data or infrastructure are incomplete, you make progress with a defensible first test while working with partners to resolve dependencies. You communicate findings and limitations clearly, make pragmatic scope decisions, and follow through until the result is useful.

Areas we are especially hiring for

We are building a team with complementary strengths. Depth in one of the four areas below can be a strong fit; we do not expect expertise in all four. We also welcome specialists and generalists in broader model and product evaluation, research engineering, and related fields. Your workstream will reflect your strengths and the team’s priorities.

Human Data and Measurement: Build reliable systems for collecting and evaluating human data. Assess whether interviews and other collection methods elicit faithful, sufficiently rich accounts of people’s experiences and behavior, and whether leading questions or unsupported detail distort the resulting data. Design rubrics, expert review, and annotation workflows; measure data quality, validity, and rater agreement; and turn these into repeatable programs that can run at scale.
Behavioral Experimentalist / Builder: Design and build creative experiments across diverse populations, settings, and behavioral tasks. Develop tools and collection workflows to run studies at scale, test whether simulations reproduce human responses to interventions, and assess how findings generalize beyond each study.
Behavioral Measurement and Causal Inference: Develop scalable strategies to learn causal relationships from large, diverse behavioral datasets. Make identification assumptions explicit and test robustness through sensitivity analyses and validation. Use this evidence to evaluate whether simulations reproduce intervention effects and support defensible counterfactual predictions across populations and settings.
Individual and Longitudinal Evaluation: Evaluate whether simulations capture stable differences between individuals and changes in their behavior over time, beyond what population averages or simple baselines predict. Build benchmarks using repeated observations, temporal holdouts, and individual trajectories. Ideally, extend this work to interactions between people, validating multi-agent behavior against observed human behavior.

Evaluations - Member of Technical Staff

The role

Evaluations - Member of Technical Staff

The role

What you will do

What you need

The core question for this role is simple

Areas we are especially hiring for

Alongside the priority areas above, we also welcome complementary strengths in

Our process

Pay and benefits

About Simile

Similar jobs at other companies

Member of Technical Staff, ML Platform (Evals)

Research, Post-Training Evals

Member of Technical Staff (Language Model Evaluations)

Researcher, Evals

Research Engineer - Evals

Member of Technical Staff - Evaluations

Interview questions

Simile

Compared with similar jobs