AI Harness & Evaluations Engineers

Capgemini

AI Harness & Evaluations Engineers Overview

Company Name Capgemini
Job Role AI Harness & Evaluations Engineers
Qualifications Bachelor’s
Category IT Jobs
Job Type Full Time
Location London

Capgemini is seeking experienced AI Harness & Evaluations Engineers to join their innovative team within the Financial Services sector. This role involves working in a hybrid environment, combining time spent at company offices, client sites, and from home, although working entirely from home is not an option. Successful candidates will undergo pre-employment checks, including identity verification, immigration status, employment history over the past three years, and a criminal record check.

Capgemini is building a new AI-native product organization focused on developing software solutions for insurance claims, payment operations, and health services, primarily sold to banks, insurers, and health plans. The engineering approach is highly autonomous: engineers define specifications, develop tooling, and create evaluation suites, while AI agents handle most implementation tasks. Human oversight remains critical, especially in regulated domains where trustworthiness is paramount.

Key Responsibilities

  • Own and maintain the evidence supporting the reliability of AI products, including high-quality datasets that define correct behavior, and ensure these datasets are developed in partnership with domain experts.
  • Manage the evaluation harness, including configuration files, specification standards, and tooling, tailoring these to your specific domain needs.
  • Design statistical gating mechanisms, such as thresholds for different data cohorts, and ensure these gates are consistently calibrated over time.
  • Integrate evaluation processes into continuous integration pipelines as merge gates, and into production systems as drift monitors that automatically score live traffic samples.
  • Analyze production failures, transforming them into regression cases to prevent similar errors from recurring silently.
  • Prepare comprehensive evaluation evidence packages that support model-risk assessments required by financial institutions, ensuring clarity for external reviewers.
  • Maintain and update local harnesses, steering files, and tooling aligned with platform standards and model development progress.
  • Work closely with claims and payments domain experts to develop domain-specific knowledge and ensure evaluation relevance.
  • Utilize AI coding assistants daily to streamline evaluation workflows and improve productivity.

Candidate Requirements

  • Hands-on experience with machine learning or data engineering, emphasizing rigorous measurement practices and working with production systems rather than just prototypes.
  • Strong statistical skills, including cohort design, sample sizing, confidence intervals, and result interpretation.
  • Experience in evaluation engineering for language models, including creating golden datasets, calibrating models as judges, implementing regression gates in CI, and understanding failure modes.
  • Proficiency in Python and experience operating within CI/CD pipelines.
  • Willingness to develop domain expertise by working daily with claims or payments specialists.
  • Experience using AI coding assistants as part of routine evaluation tasks.

Preferred Qualifications

  • Experience producing evaluation evidence accepted by external risk, audit, or regulatory bodies.
  • Experience in adversarial or red-team dataset construction, understanding safety testing boundaries.
  • Experience managing human annotation programs, including recruiting, calibration, and measuring agreement among experts.
  • Knowledge of reinforcement learning or fine-tuning pipelines driven by evaluation results and production data.
  • Published work or open-source contributions related to evaluation methodology or tooling.

Technology Stack

The role involves working with a supported platform stack including self-hosted LangSmith and LangGraph, model providers via a swappable gateway, PostgreSQL with pgvector, ClickHouse, S3-compatible storage, Neo4j Enterprise, MCP-native connectors, OpenTelemetry, Grafana, all deployed on CNCF-conformant Kubernetes with Helm and Argo CD. Experience with similar systems or components is also valuable.

Work Methodology

Engineers are responsible for writing specifications, harnesses, evaluation scripts, and guardrails, with AI agents executing the implementation. Human review is critical at key stages: specification approval, merging, and release. Critical regulated code paths such as money movement, authentication, and cryptography are always managed by humans. The team is intentionally small and senior, with quality ownership embedded within the product team. Domain experts are full-time team members, not external advisors.

Goals for Year One

  • Ensure no regression reaches clients that could have been caught by your evaluation suite.
  • Pass the model-risk review for evaluation evidence packs on first submission.
  • Address all recent production failures with regression cases.
  • Set gate thresholds based on solid evidence, fostering trust in the evaluation process for release decisions.

Additional Information

Capgemini is committed to diversity and inclusion, being a Disability Confident Employer (Level 2). The company offers a comprehensive benefits package, wellbeing initiatives, and extensive training opportunities. Employees are encouraged to innovate, learn, and grow within a collaborative environment that values making a positive impact through technology and AI.

Based in London, the role is a permanent position at an experienced professional level, with the company emphasizing a culture of continuous improvement and responsible AI development.


Degree Requirement: Bachelor’s

Visa Sponsorship May be

To apply for this job please visit careers.capgemini.com.