From test automation to AI quality engineering

From test automation to AI quality engineering

I spent many years leading test automation for large banking, government and emergency-services programs. One lesson stands out: quality is designed in, not inspected in at the end. That lesson matters even more now that organisations are putting AI assistants and agents in front of customers and staff.

Why AI is harder to test

Traditional software is deterministic: the same input gives the same output, so a test either passes or fails. AI systems are different. The same question can produce different wording each time, answers can be fluent but wrong, and behaviour can shift when a model is updated.

What carries over from test automation

  • Clear acceptance criteria – define what a good answer or action looks like before you build.
  • Automated regression suites – keep a growing set of test questions and scenarios, and run them on every change.
  • Test data management – use realistic but safe data, with personal information masked.
  • Continuous integration – quality checks run in the pipeline, not in a spreadsheet at the end.

What is new for AI

  • Evaluation sets and scoring – measure accuracy, groundedness and helpfulness against reference answers, often using a mix of automated scoring and human review.
  • Safety and red-team testing – deliberately try prompt injection, data leakage and off-topic requests to prove guardrails work.
  • Agent action testing – for agents that call tools or change systems, test every action path and confirm human approval gates hold.
  • Production monitoring – track quality signals, user feedback and drift after go-live, not just before it.

A practical starting point

Begin with 50–100 representative questions or tasks, agree what good looks like with the business, and automate the evaluation so it runs on every prompt, model or data change. That single habit catches more problems than any amount of manual spot-checking.

Quality engineering is where trust in AI is earned. If you would like help building an AI evaluation and testing approach, explore our training or get in touch.

Scroll to Top