Senior QA Automation Engineer – AI
Anaplan
Senior QA Automation Engineer – AI Overview
| Company Name | Anaplan |
| Job Role | Senior QA Automation Engineer – AI |
| Qualifications | Not Specified |
| Category | IT Jobs |
| Job Type | Full Time |
| Location | London |
Join Anaplan, a company dedicated to transforming business decision-making through innovative AI-powered planning and analysis tools. Our team is committed to delivering a leading platform used by over 2,400 global organizations, including Fortune 50 companies such as Coca-Cola, LinkedIn, Adobe, LVMH, and Bayer. We foster a culture of diversity, innovation, and achievement, where every team member’s unique qualities are valued and celebrated. Our core principles emphasize strategic thinking, values-driven actions, and disciplined execution, creating an inspiring environment for growth and success.
About the Role
We are pioneering a specialized role focused solely on quality assurance for Generative AI and agentic systems. As our Senior QA Automation Engineer for AI, you will develop testing strategies, evaluation frameworks, and quality metrics specifically designed for applications powered by large language models (LLMs). This position requires a unique combination of QA expertise, understanding of GenAI behaviors, and automation skills to ensure the AI features we develop are reliable, accurate, and trustworthy.
Your Impact
- Design and architect detailed test plans, evaluation frameworks, and quality metrics tailored for LLM-based applications and agentic workflows.
- Create and maintain comprehensive automated test suites in Python, covering API checks, UI testing, performance assessments, and evaluation procedures.
- Develop automated prompt testing systems and regression suites to detect unintended model behavior changes proactively.
- Establish multi-faceted evaluation criteria to measure AI quality across accuracy, relevance, safety, consistency, and latency metrics.
- Perform adversarial testing, including red-teaming exercises, to identify issues such as hallucinations, biases, or security vulnerabilities in AI models.
- Build internal tools and testing frameworks that enable engineering teams to validate their own GenAI implementations efficiently.
- Implement real-time monitoring and alerting systems to quickly identify and address quality issues in AI features deployed in production.
- Own and oversee the overall testing and quality assurance strategy for your workstream, maintaining high-quality test datasets and establishing best practices in QA processes.
Required Skills and Experience
- Deep understanding of Generative AI technologies, including experience with large language models, prompt engineering, and AI application design patterns.
- Strong practical experience with Python programming for scripting and automation tasks.
- Expertise with test automation frameworks such as Selenium and Pytest, and familiarity with JavaScript testing tools is advantageous.
- Solid knowledge of various testing methodologies, including functional, integration, regression, performance, and security testing.
- Proven ability to craft test cases and evaluation criteria for systems that are inherently non-deterministic like AI models.
- Excellent analytical and problem-solving skills, with meticulous attention to detail.
- Experience with API testing tools such as Postman or REST Assured, and backend testing practices.
- Experience working with CI/CD pipelines and integrating automated tests into deployment workflows.
Our Commitment to Diversity and Inclusion
At Anaplan, we believe that fostering an inclusive and equitable workplace enhances our innovation and success. We are dedicated to creating a respectful environment where all individuals, regardless of gender, ethnicity, age, disability, or background, are valued and supported. We provide reasonable accommodations for candidates with disabilities and encourage applications from diverse backgrounds.
Important Notices
Beware of fraudulent job offers circulating online. Anaplan does not send unsolicited job offers via email or conduct interviews without a thorough process involving our recruitment team. All official communications will come from an @anaplan.com email address. If you have doubts about any correspondence, contact us at [email protected].
Candidate data collected during the application process is handled in accordance with our Privacy Notice, which may include the use of AI or automated tools for evaluation purposes.
Application Process
Interested candidates can create a job alert to receive future opportunities or proceed with the application by filling out the online form, providing personal details, attaching a CV, and answering eligibility questions, including right to work and visa sponsorship requirements.
Note: The application form asks whether you will require visa sponsorship now or in the future, indicating that sponsorship consideration is possible, but the page does not explicitly confirm sponsorship availability for this role.
Degree Requirement: Not Specified
Visa Sponsorship Promising
To apply for this job please visit job-boards.greenhouse.io.