Human Data Quality Analyst, AI Business

• Run day-to-day quality measurement across live human-data programmes, combining statistical analysis with regular hands-on review to determine whether the data is fit for its intended use.
• Find the signal behind quality issues by investigating changes, quantifying their impact and distinguishing individual errors from wider problems in guidance, task design, training or tooling.
• Help build the data-quality pipeline alongside our quality engineers, turning repeated analysis into reliable validations and automated systems, including rule-based, LLM, and ML assisted approaches.
• Turn complex analysis into a clear story through reporting and visualisations that help stakeholders understand what is happening, why it matters, and what action the evidence supports.
• Support new programmes from design through to launch by helping develop the rubrics, guidelines, gold sets, calibration exercises, and production gates to measure quality from the start.
• Use what we learn from live programmes to improve annotation design, guidance, training and quality processes, then measure whether those changes work.

• You have 1 to 2 years of practical experience analysing real data in an analytical, quality, research or data-focused role.
• You can use SQL and Python to write your own queries and analysis, check whether the results make sense, and explain how you reached them.
• You can apply statistical concepts such as sampling, distributions and variability, and judge whether a result reflects a meaningful pattern or limited evidence.
• You can turn evidence into a clear account of what is happening, supported by concrete examples and practical recommendations that non-technical colleagues can act on.
• You can assess detailed work carefully and consistently while looking for patterns across the wider dataset, process or programme.
• When the right method is unclear, you can test a sensible approach, document your reasoning and recognise when you need input.

• Experience working with data produced or judged by people, such as annotation, data labelling, research coding, content review or survey research, including measures such as agreement, gold-set accuracy or defect rates.
• Knowledge of psychology, behavioural science, linguistics or a related field, particularly how people interpret meaning and make judgements.
• An understanding of how human data supports model training and evaluation, including supervised fine-tuning, reinforcement learning from human feedback (RLHF), preference data and evaluation benchmarks.
• Experience creating clear data visualisations or dashboards that help people understand and act on findings.

Why Prolific is a great place to work

We've built a unique platform that connects researchers and companies with a global pool of participants, enabling the collection of high-quality, ethically sourced human behavioral data and feedback. This data is the cornerstone of developing more accurate, nuanced, and aligned AI systems.

We believe that the next leap in AI capabilities won't come solely from scaling existing models, but from integrating diverse human perspectives and behaviors into AI development. By providing this crucial human data infrastructure, Prolific is positioning itself at the forefront of the next wave of AI innovation – one that reflects the breath and the best of humanity.

Working for us will place you at the forefront of AI innovation, providing access to our unique human data platform and opportunities for groundbreaking research.

Join us to enjoy a competitive salary, benefits, and hybrid working within our impactful, mission-driven culture. At Prolific, our compensation packages for eligible roles include base salary, equity, and benefits. Many roles also include the opportunity to earn a cash variable element, such as a bonus or commission. Your recruiter will also be happy to share the specific salary range for your preferred location during the hiring process.

Website

Youtube

Privacy Statement

By submitting your application, you agree that Prolific may collect your personal data for recruiting and global organisation planning. Prolific's Candidate Privacy Notice explains what personal information Prolific may process, where Prolific may process your personal information, its purposes for processing your personal information, and the rights you can exercise over Prolific use of your personal information.