Testing and Evaluating AI Agents — Part 1

Not long ago, I had to give a brief presentation on how to evaluate and test the development and performance of a Generative AI Agent. In today’s post, I’ll share a summary of that presentation.

History

Before diving into the specifics of the evaluation process, it’s worth taking a step back to review the broader trajectory of computational linguistics.

Machine translation of natural languages is a relatively old field of research. In the 1950s, IBM and the U.S. government invested significant resources into this area, focusing particularly on Russian-to-English translation. The earliest models were based on simple rule-based approaches, but it quickly became evident that this method would not be effective. This was widely acknowledged after the publication of the ALPAC report in 1966.

Following the report, research in computational linguistics continued at the academic level, leading to some advancements, such as the development of bag-of-words models. From the perspective of modern Generative AI, the first major breakthrough came with the introduction of word2vec models. These were the first truly practical, neural network-based approaches in the field. Their significance lay in demonstrating that language could be effectively mapped into a multi-dimensional vector space.

The next major leap was the advent of seq2seq models. These models were capable of generating outputs of varying lengths from inputs of unknown lengths, which was a significant step forward in solving real-world linguistic problems. However, they introduced two key challenges. First, there was an information bottleneck between the encoder and decoder. Second, the models were difficult to parallelize, making training extremely time-consuming. The first issue was addressed by the Attention mechanism, while the second was resolved with the Transformer architecture.

Although the Transformer architecture became widely known after the 2017 paper Attention Is All You Need, it wasn’t until November 2022 that this model type gained significant attention. This was when ChatGPT 3.5 was released, achieving immense popularity. In my opinion, this model’s qualitative leap can be attributed to two factors. First, its massive size, though size alone would not have been sufficient. The second, equally important factor was the fine-tuning of the base model using another model. This process was based on the Reinforcement Learning from Human Feedback (RLHF) method.

The release of ChatGPT 3.5 marked the beginning of the widespread adoption of today’s highly popular Generative AI solutions. Essentially, everyone is now striving to develop such systems. However, this has also brought about numerous challenges. The first is that we don’t fully understand why these models work (see Yann LeCun’s critique). While this lack of understanding might not deter business users (who often prioritize profit over comprehension), there are issues they cannot ignore. These include hallucinations, limited mathematical capabilities, biases such as racism, and the obsolescence of training data. Unfortunately, the first problem makes it difficult to address the issues in the second category.

As a result, we are currently attempting to “fortify” foundational models with various solutions. Can’t the Large Language Model (LLM) perform calculations? Then we create a calculator it can use. Doesn’t it know the latest news? We develop a search engine it can access. Can’t it access internal data? We build a Retrieval-Augmented Generation (RAG) database. AI Agents are designed to serve these purposes.

Over time, we’ve been creating increasingly complex systems. The current solutions are built on a highly flexible yet not fully understood Generative AI model. This is promising enough to attract significant investments into the field, but it’s not yet mature enough to immediately convert those investments into profit. With AI Agents, we are constructing ever more intricate systems around foundational models to finally make these solutions profitable.

However, this presents new challenges. Testing and evaluating a flexible and complex system is far more complicated than, for example, applying a unit test or a confusion matrix. In the remainder of this post, we’ll explore how to address this problem effectively.

Developing AI Agents

Developing an AI Agent is fundamentally similar to building a traditional machine learning model. Essentially, it involves a cyclical process of development and monitoring.

During the training phase, we run the Agent on a test dataset and evaluate the results. Based on the evaluation, we refine the model and test it again. This process is repeated until we deem the model ready for deployment to actual users.

The live system is continuously monitored. If abnormal model performance is observed, the input data causing the issue is saved and added to the test dataset in the training environment.

This process can be illustrated as follows:

Source: Arize

The above process is fundamentally similar to what we use for other machine learning models, but there are some unique challenges specific to AI Agents. The first challenge is that Generative AI models are non-deterministic. While many machine learning models are not entirely deterministic during the training phase (e.g., the initial weights of neural networks can be randomly initialized), the results of a trained model are typically deterministic. Unfortunately, this is not the case for Large Language Models (LLMs). Even trained models do not reliably produce the same output (unless the temperature parameter is set to zero, but this can lead to overfitting). As a result, the deterministic evaluation and testing methods we’ve traditionally used are not effective for these models.

LLM as a Judge

One option for evaluating the performance of our model is to use a Large Language Model (LLM) as a scoring mechanism. The essence of this approach is to craft a prompt that instructs the LLM to determine whether a given text meets certain criteria. The obvious advantage of this method is its flexibility. Since we are using an LLM for scoring, we have significant freedom in designing the test and can handle input flexibly.

Personally, I encountered this method in 2023 when I began developing Agents that, in their final step, compared the original question with the proposed final answer and then scored whether the answer truly addressed the question. If it didn’t, the Agent restarted the entire process. The instruction used for this was as follows:

You are given a question and an answer. You must determine whether the given answer answers the question. Here is the data:

[Question]: {input}

[Answer]: {output}

Your response must be a single word, either “correct” or “incorrect”, and should not contain any text or characters aside from that word. “correct” means that the question is correctly and fully answered by the answer. “incorrect” means that the question is not correctly or only partially answered by the answer.

This step was one of the components of the Agent itself in my implementation, but the technique can be generalized and applied outside of the Agent for performance evaluation.

However, there are a few important considerations when using Large Language Models (LLMs) as judges. First, as mentioned earlier, these models are non-deterministic. This means that they can produce different responses to the same input at different times. Therefore, it is recommended to run the test multiple times and use some form of aggregation for the final evaluation. For example, in the case of the example above, you could use the result that appears most frequently. If the instruction is run three times and the results are “correct,” “incorrect,” and “incorrect,” the final evaluation would be “incorrect.”

Second, LLMs are not particularly well-suited for continuous scoring. The variance in responses tends to be much higher when asking for a percentage score compared to when asking the model to choose from a limited set of categories. It’s advisable to minimize the number of possible categories the model has to decide between. Let’s look at an example!

The Arize team conducted an interesting test for their blog post Why You Should Not Use Numeric Evals for LLM As a Judge. In three different scenarios, they used a Large Language Model as a judge: grading spelling, detecting frustration, and identifying negative sentiment. In each case, the model was asked to evaluate a text on a scale from 0 to 10, which had already been scored by experts. Let’s see how ChatGPT 4 performed in this test:

In the ideal scenario depicted in the above chart, the data points would align along a straight line running from the bottom-left corner to the top-right corner. This would indicate that the model and the human expert assigned identical scores to the texts. As we can see, the results are quite far from this ideal case. ChatGPT 4’s scoring for spelling reminds me of the Hungarian language teacher I imagined as perfect during my childhood: it only refrains from giving excellent grades in the most extreme cases of poor performance.

Third, not all Large Language Models are created equal. In the blog post mentioned earlier by the Arize team, the same tests were conducted using multiple models:

As shown in the chart, the performance of different models varied significantly.

Of course, the fact that we can use an LLM to score our Agents doesn’t mean it’s our only option. We can certainly experiment with other, more quantifiable methods, but it’s important to ensure that these methods are flexible enough. For instance, the semantic similarity of texts can be effectively measured using Cosine distance. This can be particularly useful when we know the expected output text, as it allows us to easily measure and score the distance between the actual response and the expected text.

Where is the Value?

I understand that the ultimate goal is the correctness of the final response, but it’s also worth considering how much value we are actually adding when working with an Agent. This perspective is important to discuss in the context of this topic.

The development of current Large Language Models (LLMs) is extremely resource-intensive, which means that relatively few individuals and companies will be creating such models. For most developers, the key question won’t be how the Agent’s performance changes if they switch from OpenAI’s model to DeepSeek’s. The reason is that hundreds of developers and companies will be using the same models. What’s far more important is understanding how much value we can add to these models, as this determines our competitive advantage — or disadvantage. For this reason, it’s crucial to measure this aspect as well.

The majority of a company’s added value typically comes from three sources:

1. Data: Proprietary, non-public data that the model can work with.
2. Instructions: Optimized prompts crafted by developers.
3. Architecture: The design and operational structure of the Agent.

It’s essential to evaluate the performance of Agents whenever any of these three factors are modified.

Closing Thoughts

In the next part, we’ll explore how the evaluation of AI Agents can be implemented on the Phoenix platform.

Originally published at http://sajozsattila.home.blog on July 20, 2025.