AI Research Scientist

The role

The AI Research Scientist works at the intersection of AI, psychometrics and research, ensuring that Maki’s automated scoring systems are scientifically robust, fair, and continuously improving.

Wait…

Evaluating and testing LLMs for scoring
Assess the accuracy and reliability of LLMs used for automated scoring, such as structured grids, job-specific skills and multilingual proficiency (written and spoken).
Benchmark closed- and open-source models across assessment types and languages.
Test robustness to prompt variation, input noise and adversarial behaviour such as prompt injection.
Explore prompt engineering and model optimisation, and recommend which models and prompts go into production.
Building our own scoring models
Move our scoring onto self-hosted open-weight models, and fine-tune them on expert-rated data where they can beat frontier models.
Check every model switch keeps accuracy, reliability and fairness at least as good as today's.
Comparing AI with human judgement, and calibrating psychometric models
Design studies that compare AI scores with expert ratings, against both individual experts and expert averages, to establish agreement and validity.
Develop hybrid scoring pipelines with engineering that combine human oversight with AI scoring.
Evaluate and calibrate psychometric models for validity and comparability across populations and test forms.
Bias and fairness analysis
Detect and analyse bias in AI-generated and psychometric scores across demographic groups, including subgroup analysis.
Apply mitigation techniques such as reweighting and calibration without losing model performance.
Contribute to fairness dashboards and compliance documentation that support transparent model governance.
Predictive analytics and performance insights
Model the relationship between candidate scores and job performance and retention, using large assessment and performance datasets.
Develop and test models that estimate success probabilities or identify behavioural and linguistic predictors of performance.
Work with data science, implementation and customer success teams to turn findings into recommendations for clients and colleagues.
Research output
Publish reports, papers and conference presentations, internal and external.
Explain technical findings clearly to non-technical colleagues and clients.

AI Research Scientist

The role

AI Research Scientist

The role

The team

What you need

Nice to have

Application process

Similar jobs at other companies

Research Scientist (Remote/US/LATAM)

AI Research Manager - Meta Superintelligence Labs

Research Scientist

Member of Technical Staff (Language Model Evaluations)

Machine Learning Scientist - ML Research

AI Evaluation Scientist

Interview questions

MakiPeople