I Thought the Hard Part Would Be the AI. It Turned Out to Be the JSON.

Over the last few months, I have been spending time exploring how AI can be used in test automation workflows. Like many others, I have seen plenty of examples where an LLM can explain code, generate tests, or summarize logs. I wanted to see whether it could help with something I deal with regularly: understanding why automated tests fail.

The idea sounded simple enough.

A Playwright test fails. Instead of opening reports, reading stack traces, and digging through logs, an LLM should be able to look at the failure information and provide a quick explanation of what went wrong.

To experiment with this, I built a small proof of concept using Playwright, GitHub Actions, and an LLM.

The goal was not to build a production-ready framework. I simply wanted to understand what would happen if test failures could be translated into something easier to consume.

At first, I thought the biggest challenge would be the AI itself. I assumed I would spend most of my time refining prompts and figuring out how to get useful responses from the model.

I was completely wrong about that.

The first step was straightforward. I intentionally introduced an incorrect locator into a login test. Instead of targeting the correct username field, the test looked for a locator that didn’t exist. As expected, the test failed and eventually timed out.

Playwright captured everything I needed. The report contained stack traces, screenshots, execution details, traces, and a JSON results file. From a testing perspective, all the information was there.

The problem started when I tried to extract the failure and send it to the LLM.

For some reason, my script kept telling me that no failures were found. The tests were failing, the reports were available, and the evidence was sitting right in front of me. Yet the parser behaved as though everything had passed.

My parser reported “No failures found” even though Playwright had generated failure results.

That didn’t make any sense. GitHub Actions showed failed executions and the Playwright report clearly showed failed tests, but my extraction logic was behaving as if everything had succeeded.

My first instinct was to investigate the AI integration. Maybe the prompt was not working. Maybe the model was missing something.

After spending more time with the results file, I realised the issue had nothing to do with AI.

The real problem was my assumption about the structure of the Playwright JSON report.

I expected the failure information to be easy to locate. Instead, it was buried several levels deep inside nested result objects. The statuses were also different from what I initially expected. Rather than a simple “failed” value, the report contained statuses such as “timedOut” and “unexpected”.

Because my extraction logic was looking for the wrong structure, the LLM never received the failure details in the first place.

Once the extraction logic was corrected, the actual failure information could be passed to the LLM.

That was a surprisingly useful lesson.

We often talk about AI as if the model is the centre of the solution. In reality, the quality of the outcome depends heavily on the quality of the context we provide. If the extraction layer fails, even the most capable model becomes irrelevant.

Once I corrected the parser and passed the actual failure details to the LLM, the results became much more useful.

The model identifies the likely failure category and explains the probable root cause.

The model identified that the test was waiting for a locator that didn’t exist.

The generated analysis also suggested possible investigation and remediation steps

It classified the issue as a locator problem, explained why the timeout occurred, and suggested areas to investigate. It was not performing magic, and it was not replacing debugging. What it was doing was reducing the time required to reach an initial hypothesis.

That is where I started seeing practical value.

Instead of scrolling through pages of logs, I could get a concise summary of the likely root cause and then continue my investigation from there.

The most valuable takeaway from this project was not that AI can analyze test failures. Many engineers already know that.

The bigger lesson was that AI integrations succeed or fail based on the data pipeline feeding them. Before thinking about prompts, models, or agents, it is worth asking a simpler question: Are we actually providing the model with the right information?

In my case, a single incorrect assumption about a JSON structure was a bigger obstacle than the AI itself.

And that wasn’t the lesson I expected to learn when I started the project.