AI Harness Engineers

  • Full Time
  • London

Capgemini

AI Harness Engineers Overview

Company Name Capgemini
Job Role AI Harness Engineers
Qualifications Bachelor’s
Category IT Jobs
Job Type Full Time
Location London

In this role, you will be responsible for ensuring the reliability of AI products within a regulated domain by managing evaluation evidence for a specific product line. Your work will involve creating and maintaining golden datasets that define correct behavior, designing gates to prevent regressions, and implementing drift monitors to maintain quality in production.

Key Responsibilities

  • Own the evaluation evidence for a specific product line, ensuring the integrity of golden datasets that define correct behavior.
  • Design gates to prevent regressions from reaching clients, utilizing statistical methods and honest sample sizes.
  • Implement drift monitors that automatically score a sample of live traffic to maintain quality in production.
  • Create a failure flywheel to analyze production failures and establish permanent regression cases to prevent recurrence.
  • Develop an evaluation evidence pack for customer model-risk validation, ensuring it is comprehensible to non-engineering stakeholders.
  • Maintain the local harness, including steering files and specification conventions, ensuring alignment with platform patterns.

Requirements

  • Experience in machine learning or data engineering with a strong focus on measurement and production systems.
  • Proficient in statistics, including cohort design, sample sizing, and confidence intervals, with the ability to critically assess results.
  • Hands-on experience in evaluation engineering for language models, including the creation of golden datasets and regression gates.
  • Strong programming skills in Python and familiarity with CI/CD systems.
  • Willingness to develop domain knowledge by collaborating closely with claims or payments experts.
  • Daily use of AI coding assistants as part of the workflow.

Benefits

  • Flexible working arrangements, including a hybrid model of office and remote work.
  • Access to a wide range of training and development opportunities, including certifications from recognized institutions.
  • Wellbeing initiatives, including mental health support and access to wellbeing apps.
  • Peer recognition programs to celebrate employee achievements.
  • A commitment to inclusive recruitment practices, particularly for candidates with disabilities.

Degree Requirement: Bachelor’s

Visa Sponsorship May be

To apply for this job please visit careers.capgemini.com.