QA Automation Engineer – AI
Anaplan
QA Automation Engineer – AI Overview
| Company Name | Anaplan |
| Job Role | QA Automation Engineer – AI |
| Qualifications | Not Specified |
| Category | IT Jobs |
| Job Type | Full Time |
| Location | London |
Anaplan is seeking a dedicated QA Automation Engineer specialized in AI to join their innovative team in London. This role focuses on ensuring the quality and reliability of GenAI and agentic systems, particularly large language models (LLMs), conversational AI, and related workflows. As part of this position, you will develop and execute automated testing strategies, create evaluation frameworks, and measure AI performance across various metrics such as accuracy, relevance, safety, and latency.
Key Responsibilities
- Design and implement comprehensive automated testing plans for AI features, including conversational and agentic systems.
- Build and maintain automated test suites that cover APIs, user interfaces, and performance benchmarks using Python scripting.
- Develop evaluation frameworks and metrics to assess AI performance, ensuring models meet quality standards.
- Curate diverse test datasets and ‘golden examples’ that reflect real-world user scenarios and edge cases.
- Perform adversarial testing to detect issues like hallucinations or biases, and support the integration of alerting systems for production monitoring.
- Work closely with developers to facilitate validation of AI code through internal testing tools and frameworks.
- Participate in user acceptance testing to verify that AI features function as intended and meet user needs.
- Document testing procedures, coverage, and known issues clearly and thoroughly for team reference.
- Contribute to ongoing improvements in testing methodologies and best practices for AI quality assurance.
Required Skills and Experience
- Proven experience in QA or test engineering, with a particular interest or background in AI/ML systems.
- Proficiency in Python programming, including scripting and automation frameworks such as Pytest.
- Strong understanding of GenAI technologies, including LLMs, prompt engineering, and AI application patterns.
- Experience with test automation tools and scripting for functional, integration, regression, and security testing.
- Ability to develop and execute test cases for non-deterministic AI systems, ensuring consistent evaluation.
- Familiarity with API testing tools like Postman or REST Assured, and backend testing processes.
- Knowledge of CI/CD pipelines and automated testing integration within development workflows.
- Solid grasp of software testing principles, evaluation metrics, and quality assurance best practices.
Our Commitment to Diversity and Inclusion
At Anaplan, fostering an inclusive culture is fundamental to our success. We value diversity of thought and encourage individuals from all backgrounds to bring their authentic selves to work. We are committed to providing reasonable accommodations for candidates with disabilities and ensuring an equitable hiring process. Our environment promotes respect, collaboration, and continuous development, making it an ideal place to grow your career in AI and quality assurance.
Important Notices
We advise candidates to be cautious of fraudulent job offers circulating online. Genuine offers are made only after a thorough interview process, and all official communications will come from an @anaplan.com email address. If you have any doubts about the authenticity of a communication, contact us at [email protected].
Degree Requirement: Not Specified
Visa Sponsorship May be
To apply for this job please visit job-boards.greenhouse.io.