Stop reading CI/CD logs. Let AI do it

Over 80,000 lines. One root cause. How I used local AI to fix Jenkins log analysis. A POC using Llama 3.1, BART, and smart preprocessing (regex) .

A build fails. It’s 3 PM on a Friday. Jenkins opens — and we may see a log that’s 80,000 lines long. Somewhere inside, maybe on line 47,000 😄buried inside a snapshot revert stage nobody touched this week, is the one line that explains why the pipeline is on fire. So I scroll, Ctrl+F for ‘ERROR’. Six hundred results. Sigh!! 😟

That used to be my situation every single time. So I built a small POC to fix it — and along the way, it taught me more about how LLMs actually work.

Below is the component connect diagram.

How the components connect — YAML holds the rules, Python runs the engine, Ollama keeps it local

Streamlit: It’s a Python library that turns a script into an interactive web UI — we write st.metric() or st.chat_input() and it renders a live page in the browser, no HTML or JavaScript needed. There’s no React frontend, no Flask API, no separate backend. The entire interactive UI — file upload, metrics dashboard, extraction details, chat interface — is built in the same Python file as the logic, using Streamlit.

Ollama: Think of it like Docker, but for AI models — it lets us pull and run large language models locally with a single command, no cloud account needed. The python code talks to it over localhost, the model runs entirely on local machine, and no data ever leaves.

The Real Problem With CI/CD Logs

Jenkins logs aren’t just long. They’re structurally chaotic. A typical failure log has several stages that passed fine, one stage that actually failed, five stages that got skipped because of that failure, and thousands of lines of pure noise — environment variable dumps, sshretry warnings, pip root user warnings, docker credential messages, and tool-level debug logs that tell us nothing useful.

If we dump all of this into an LLM and say “what’s the root cause?”, two things happen. First, the model probably can’t fit it all in memory. Second, even if it could, the response will be vague and occasionally hallucinated — because the model is trying to reason across 800,000 tokens of mostly irrelevant text.

The answer isn’t a bigger model. It’s smarter preprocessing.

The Architecture: Six Layers Before the LLM Sees Anything

The Six-Layer RCA Pipeline — from raw log to structured answer

Here is how the tool works, from raw log all the way to the final answer.

Layer 1: Stage Parser — Scope to the failure first

A Jenkins pipeline is structured into named stages — Checkout, Build, Test, Deploy. Each has a start and end marker in the log. The Stage Parser reads the whole log and builds a simple map. The moment it finds the first failed stage, only those lines move forward. An 80,000-line log might have just 1,200 lines inside the failed stage. That’s a 98.5% reduction — before any AI is involved at all. This matters

Layer 2: Noise Filter — Remove the clutter

Even inside the failed stage, there’s a lot of noise. The filter removes environment variable dumps, SSH retry messages, pip warnings, Docker credential messages, and tool DEBUG/INFO lines. All noise rules live in a patterns.yaml file — no Python changes needed, just add a regex pattern to the YAML.

Layer 3: Regex Engine — Categorize the errors

With clean, meaningful lines from the failed stage, the tool runs them through error category patterns — all defined in the same YAML. Categories like:

Each category collects its matching lines, strips timestamps and ANSI color codes, and deduplicates them.

Layer 4: Priority Sorter — Send only what really matters

We might end up with 8 matched categories. They get sorted by priority and only the Top K (default: 3) move forward. Everything else is shown on screen for transparency but never sent to the AI.

This is one of the most important insights in prompt engineering: the quality of LLM output depends on the quality of the input context — not the size of the model.

Layer 5: Prompt Builder — Tell the AI exactly what to do

The assembled context becomes a structured message. The system prompt sets the role, specifies the exact output format (root cause, failed stage, key error lines, recommended fix, cascading impact), adds strict anti-hallucination rules, and injects one few-shot example to show the model what good output looks like.

Layer 6: Llama 3.1 via Ollama — Get the answer

The prompt goes to Llama 3.1 8B, via Ollama. Temperature is set to 0.1 — near-deterministic, consistent output.

The RCA and recommended fix we got from streamlit UI

What Is a Token, and Why Does It Matter?

A token is not a character and not a word. It’s a small chunk of text the model reads one piece at a time.

Simple rule: 4 characters ≈ 1 token.

Every LLM has a context window — the maximum tokens it can process at once. Llama 3.1 has a 128k token window. An 80,000-line log is roughly 1 million tokens. It doesn’t fit. And even if it did, asking a model to reason across 1 million tokens of mostly irrelevant text is like asking someone to spot a typo in the entire works of Shakespeare.

The preprocessing pipeline brings that down to ~750 tokens. Less than 0.1% of the original. The AI sees a surgical extract — and gives a precise answer.

The UI shows this live:

Total: 2,074 chars ≈ 528tokens. That’s not just a display number. It’s a health check. If it creeps above 4,000–5,000 tokens, the filters are letting too much through.

BART: The model that catches when pipeline lies

There’s one scenario the main tool doesn’t handle on its own. The pipeline reports ‘Finished: SUCCESS’ — but the log actually contains hidden errors. A stage silently catches an exception. A deployment script fails internally but the pipeline step still marks itself green. This happens more often than people expect. That’s where BART comes in.

BART (Bidirectional and Auto-Regressive Transformer, specifically facebook/bart-large-mnli uses zero-shot classification — classifying text into categories without training on our own data. When the pipeline reports SUCCESS, BART receives the last 50 lines of the log plus any suspicious lines (WARN, timeout, failed, error — but not DEBUG or INFO). It classifies those lines against four labels:

• build failure
• test failure
• warning only
• successful build

Suppose BART returns:

build failure: 0.72 ← 72% confident test failure: 0.18 warning only: 0.07 successful build: 0.03

A threshold check runs (default 60%). 72% > 60% and it’s a failure type → hidden errors detected → full RCA pipeline runs.

BART is the gatekeeper. Llama is the analyst. Two different models, two different jobs, one pipeline.

Why Vector Search wasn’t the right fit for my current use case— and when It would be?

A vector search version was built and tested. Then it was set aside as the primary approach.

The problem: CI/CD error messages are structured, predictable text. An error like snapshot-service — ERROR — Failed to revert: database relation does not exist doesn’t need semantic understanding. It needs a regex. There’s no ambiguity that requires a 768-dimensional vector space to sort out. Vectorization adds complexity and non-determinism for zero benefit on structured log text.

But vector search has a real future here — for historical search. Right now, every build failure is analyzed in isolation. With FAISS historical indexing, after every failure the extracted error lines get embedded and stored with metadata.

An engineer asks: When did we last see a database error in the Snapshot Revert stage?

FAISS returns the 10 most similar past failures. The LLM summarises:

This stage has failed 3 times in 6 weeks, always due to database schema issues.

Regex and FAISS aren’t competing approaches. Regex extracts the right lines. FAISS remembers them.

The Bottom Line

Not just Jenkins — Any build tool works

While Jenkins is the example throughout this article, this architecture is not limited to Jenkins at all.

Whether logs come from GitHub Actions, GitLab CI, Bitbucket Pipelines, or any other build tool, the underlying problem is exactly the same — massive unstructured log output with the real error buried somewhere inside.

• GitHub Actions has Jobs and Steps instead of Stages
• GitLab CI has Pipelines and Jobs

All of them produce environment variable dumps, retry warnings, and tool debug logs that need to be filtered out. Because the tool is fully configuration-driven, adapting it to GitHub Actions or GitLab CI means updating the patterns.yaml file to match their specific step markers and error formats. The LLM pipeline, noise filtering, BART integration, and FAISS support stay exactly the same.

At its core, this is a disciplined text processing pipeline that feeds a carefully constrained LLM the smallest possible piece of evidence it needs to do one job well. The AI doesn’t read the log — it reads a surgical extract of 750 tokens out of a possible million.

BART catches the lies, YAML defines the rules, regex finds the evidence, Llama explains the failure and Ollama keeps it all private.

References:

[embed]BART · Hugging Face We're on a journey to advance and democratize artificial intelligence through open source and open science.huggingface.co

[embed]*Streamlit A faster way to build and share data apps* Streamlit is an open-source Python framework for data scientists and AI/ML engineers to deliver interactive data apps …*streamlit.io

[embed]Industry Leading, Open-Source AI | Llama Discover Llama 4's class-leading AI models, Scout and Maverick. Experience top performance, multimodality, low costs…www.llama.com

Thank you for reading!!