AI Evals: Where QA Skills Meet Generative AI

I have been following the concept of AI Evals since last year and the more I get to know about it the more I lean towards the perspective….that’s QA work. Yes it is for PMs who want to make sure their AI agent’s performance is up to the mark but the approach needed for Evals looks similar to “black box testing” where we evaluate and grade the AI outputs without knowing how exactly it was processed, generated or the logic behind it.

I had created a chatbot for my friend’s fitness app so I decided to perform an AI eval and dig more into how this can align with QA principles. The chatbot was for a fitness app where the clients can manage workout classes, tracks progress, and provide information.

Step 1 : Set up Eval criteria

Eval criteria are basically metrics that help us analyse whether the AI output is aligned with our needs. Think of it as a quality check for AI. Instead of just asking, “Did it give an answer?”, you use these criteria to ask : “Was the answer correct, safe, unbiased, and helpful?”

For a Fitness related chatbot I decided to measure it’s output based on metrics like accuracy (does it have correct knowledge about the product/services), tone, relevance of follow up questions and user engagement. And I defined the scores for Good, Avg and Bad performance ⬇️

Eval criteria

Step 2 : Golden dataset

A golden dataset is a carefully curated set of test inputs with trusted “correct” outputs used to evaluate the chatbot. It acts as a benchmark to consistently measure performance across the eval criteria (accuracy, tone, etc.) Once this golden dataset is ready we can use it for manual eval/human grading (aka manual testing 😁) and later on for LLM-as-a-judge evals (aka automated AI tests 🤓)

Here is one of the examples from the golden dataset I created. Google Sheet link here

Example of golden dataset

Step 3 : Manual labelling/Human Grading/Manual testing😉

Manual labeling/Human Grading is the process where humans evaluate chatbot responses using predefined eval criteria. We basically read each user input and AI generated reply, then assign scores (e.g., accuracy, tone, engagement). It ensures high-quality, reliable ground truth for the golden dataset. It is very useful for catching nuance (emotion, safety, context) that automated systems may miss.

However, if there are no proper guidelines set it may lead to the most boring time-consuming thing you do with AI output. To avoid this, decide and refine the user input queries to a minimum range of 5 to 10. Make sure to cover most asked + important aspects of your AI tool. For example, in my case I covered 5 user inputs, each of them covering different aspects like booking a training session, trainer availability, workout trackers, etc.

In below screenshot you can see the scores I assigned for each of the AI generated responses, and added Notes for further improvements to the chatbot.

Manual Eval

Step 4 : LLM-as-a-judge eval

LLM-as-judge evaluation uses a language model (mostly gpt) to automatically score chatbot responses based on eval criteria. Basically we ask/prompt the model to act as an evaluator and assign scores. This allows for fast, scalable, and consistent evaluation across large datasets. However, it depends on prompt quality and may inherit biases or errors from the judging model.

The most popular tools for such kind of evals are LangSmith, MLFlow and some also use Claude Console!? and if you want open source tool I came across LangFuse.

All these tools are pretty awesome for evals but you will need an API key for the models you would like to use. The most common model used for evals is gpt (mini or any other) to which you will need to have a subscription for obtaining its API key. Currently I don’t want to dive into subscriptions to AI tools so I’m not including this step in detail. But it’s basically running automated regressions with the help of our golden dataset.

In conclusion, I think AI evaluation naturally aligns with the mindset and skills of QA and SDET professionals because, at its core, it is still a form of quality assurance.

QA/SDET already possess the skills needed for AI evals : designing test scenarios and edge cases, defining evaluation criteria, identifying regressions, building automated validation pipelines and thinking from an end-user perspective.

As AI applications become more common, us QA engineers and SDETs are uniquely positioned to evolve into “AI Quality Engineers” who ensure not only functional correctness, but also response quality, safety, and reliability. AI evals may not replace traditional testing, but they are increasingly becoming a natural extension of the modern QA skillset.

Let me know what you think about this perspective!