SDET – AI

Anaplan

SDET – AI Overview

Company Name Anaplan
Job Role SDET – AI
Qualifications Not Specified
Category IT Jobs
Job Type Full Time
Location London

Join a forward-thinking team at Anaplan, where innovation and AI-driven solutions are at the core of our business. We are seeking a Senior SDET specializing in Artificial Intelligence to lead quality assurance efforts for our GenAI and agentic systems. In this role, you will be responsible for designing and implementing testing strategies, evaluation frameworks, and quality metrics specifically crafted for large language model (LLM) applications. Your work will ensure that our AI features are reliable, safe, and perform optimally across various parameters.

Key Responsibilities

  • Develop and architect detailed test plans, evaluation frameworks, and quality metrics tailored for LLM-powered AI applications and workflows.
  • Create and maintain comprehensive automated test suites in Python, covering evaluation procedures, API validation, user interface testing, and performance assessments.
  • Build automated prompt testing systems and regression suites to detect unintended changes in AI model behavior proactively.
  • Establish multi-faceted evaluation criteria to measure AI quality in terms of accuracy, relevance, safety, consistency, and latency.
  • Perform adversarial testing, including red-teaming exercises, to identify issues such as hallucinations, biases, or security vulnerabilities within AI models.
  • Develop internal tools and testing frameworks that enable engineering teams to validate their AI implementations efficiently.
  • Implement real-time monitoring and alerting systems to quickly identify and address quality issues in production AI features.
  • Maintain ownership of the overall quality and testing strategies, including managing high-fidelity test datasets known as “golden examples” and establishing QA best practices.

Required Skills and Experience

  • Extensive understanding of Generative AI, including large language models, prompt engineering, and AI application design patterns.
  • Hands-on experience with Python programming for automation and testing purposes.
  • Proficiency with test automation frameworks such as Selenium and Pytest, along with scripting in JavaScript if needed.
  • Solid knowledge of various testing methodologies including functional, integration, regression, performance, and security testing.
  • Ability to design test cases and evaluation metrics for systems that do not produce deterministic outputs.
  • Strong analytical and problem-solving skills with meticulous attention to detail.
  • Experience with API testing tools like Postman and REST Assured, and backend testing processes.
  • Familiarity with CI/CD pipelines and integrating automated testing into continuous development workflows.

Our Commitment to Diversity and Inclusion

At Anaplan, we believe that fostering an inclusive environment enhances our innovation and success. We are dedicated to attracting and retaining diverse talent, respecting individuals regardless of gender, ethnicity, age, disability, neurodiversity, or other characteristics. We ensure reasonable accommodations are provided for candidates with disabilities during the application and interview process, and throughout employment. Our culture values authenticity, collaboration, and continuous growth, making us a place where everyone can thrive.

Important Notices

We advise candidates to be cautious of fraudulent job offers circulating online. Genuine offers are only extended after a thorough interview process involving our recruitment team and hiring managers, typically via video or in person. All official communication from Anaplan will originate from an @anaplan.com email address. If you have any doubts about the authenticity of any correspondence, please contact us at [email protected].

Candidate data collected during the application process is handled in accordance with our Candidate Privacy Notice. We may use AI or automated tools to assist in evaluating applications.


Degree Requirement: Not Specified

Visa Sponsorship May be

To apply for this job please visit job-boards.greenhouse.io.