Lead AI Tester / QA Lead (LLM & GenAI)

Job description

We’re supporting a consulting partner on the search for a Lead AI Tester / QA Lead to define and own the testing strategy for several AI / LLM use cases at a major financial services client. This role is primarily about strategy, governance and delegation, with some scope to remain hands‑on.

The opportunity

You will:

• Create and maintain the test strategy and playbook for AI solutions - covering functional and non‑functional testing, regression, safety, bias, hallucinations, performance and cost.
• Translate that strategy into clear documentation and workflows (e.g. Confluence playbooks, Jira ticket structures, test plans and checklists).
• Delegate and coordinate execution across QA engineers and developers, ensuring test activities are properly defined, prioritised and completed.
• Define how AI testing fits into the CI/CD pipeline and SDLC: where tests run, what they block, and how results are surfaced.
• Establish and track evaluation metrics and monitoring for AI systems (e.g. Quality, drift, safety, latency, cost), and report these to stakeholders.
• Guide and engage both internal and client stakeholders on all aspects of testing – test strategy definition, standards, metrics, monitoring and release readiness.

What we’re looking for

Must have:

• Proven experience as a Test Lead / QA Lead / SDET owning test strategy and automation in complex environments (ideally financial services or similar).
• Ability to create a test strategy document or playbook for AI solutions and keep it current.
• Experience delegating execution of a test strategy to other team members – for example via Jira ticket definition and structured test packs.
• Strong understanding of how tests and environments fit into CI/CD pipelines and wider SDLC activities.

Should have:

• Practical exposure to testing AI / ML / LLM‑based systems (e.g. Chatbots, RAG applications, AI‑assisted workflows) and an interest in deepening this.
• Familiarity with approaches to evaluating LLM outputs (for example, semantic‑similarity based methods or using LLM‑as‑a‑judge), and with concepts such as hallucinations, safety, and drift.
• Experience working in consulting or client‑facing roles and presenting testing approaches and results to senior stakeholders.

Details

• Location: UK‑based, hybrid working with client and consulting teams.
• Contract: Initially 3–6 months, with strong potential to extend.
• Day rate: Up to around £600–650 per day, depending on experience.

If you’re a QA leader who wants to shape how AI and LLM solutions are tested in a regulated environment and can turn that thinking into a clear, usable playbook, please get in touch.

Extra information

Status

Open

Education Level

Secondary School

Location

London Area

Type of Contract

Full-time jobs

Published at

20-05-2026

Profession type

Accountancy

Full UK/EU driving license preferred

No

Car Preferred

No

Must be eligible to work in the EU

No

Cover Letter Required

No

Languages

English