We Let AI Fix Security Vulnerabilities in Production. Here Is What Actually Happened.

An AI-generated fix is a hypothesis until it survives the gates: tests, re-scan, human review.

A practitioner’s account of building an AI-assisted security remediation loop for a multi-tenant SaaS platform: the noise problem, the trust problem, and the architecture that solved both.

Everyone is writing about AI in the software development lifecycle. Most of it is written by people who have never had to decide whether an AI-generated fix is allowed to merge into a production branch that fourteen enterprise tenants depend on.

I have had to make that decision. This is the story of how we made it, what we built, the numbers it produced, and the mistakes we made along the way.

The environment

The platform in question is a multi-tenant SaaS product running on Kubernetes across two clouds. The estate is what you would expect from a decade of real product development: 35 repositories, 45 container images, 9 public-facing domains, and a polyglot codebase spanning .NET, Python, JavaScript and TypeScript, and Go, plus all the Terraform and Helm that holds it together.

Multi-tenancy raises the stakes in a way that is easy to say and hard to feel until you operate one. A misconfigured pipeline is not one company’s incident. It is every tenant’s incident. So the security bar was already high before AI entered the picture.

The challenge was never detection

Here is the uncomfortable truth about modern application security: finding vulnerabilities is a solved problem. We had layered scanners covering static analysis, dependencies, infrastructure code, container images, secrets, dynamic testing, and cloud posture. They worked. They worked so well that they were burying us.

In one measurement period, the scanners produced 760 raw findings. Read that number as an engineering manager and you feel the dread. Because you know what most of those findings are: duplicates reported by three different tools, vulnerabilities in code paths that never execute, false positives that will consume an hour of investigation each before being dismissed.

Engineers were spending their days sorting noise instead of fixing real problems. And while triage consumed the team, the genuinely dangerous findings aged. Known CVEs in dependencies sat unresolved, not because anyone decided to ignore them, but because the queue in front of them was full of garbage.

There was a third problem, quieter and more corrosive than the other two. When a fix did ship, we had a ticket saying so and nothing more. A status update that reads “resolved” is a claim. It is not evidence. Nobody could say with certainty that a fix had actually closed the finding without breaking something adjacent.

Three challenges, then. Too much noise. Too little remediation capacity. And no structural proof that fixes worked.

The resolution: a loop, not a tool

The temptation with AI security is to buy something and point it at your repos. We took a different view. The change that mattered was architectural. We built a loop with four stages, and the design principle that governs all of it fits in one sentence:

AI does the fixing. The pipeline does the proving.

Stage one: detect everywhere, on every pull request

Layered scanning stayed. GitHub Advanced Security for code and secret scanning. A commercial security platform covering SAST, dependency analysis, infrastructure code, containers, secrets, DAST, and cloud posture. Trivy on images. Code quality scanning and npm audit in CI. Detection runs on every pull request, every day.

Nothing revolutionary here, and that is the point. Detection is table stakes. The interesting work starts after detection.

Stage two: let AI kill the noise before humans see it

This is the stage where most security programs die, so it is the stage where AI earned its keep first.

An AI triage layer now deduplicates findings across scanners and suppresses the ones that do not matter: duplicates, unreachable code paths, false positives. In our measurement period, 549 of the 760 raw findings were suppressed before any engineer looked at them. That is 72 percent of the noise, gone. The platform valued the saved effort at roughly 277 engineering hours.

What remained was a queue of 211 grouped issues, ranked by exploitability rather than raw CVSS score. That distinction matters more than any other tuning decision. A critical-severity finding in dead code is less urgent than a medium in your authentication path, and a triage system that cannot tell the difference is just a louder scanner.

Stage three: remediate in two lanes

Dependencies first, because they are the volume problem. Automated upgrade pull requests are raised continuously. If a PR passes the full validation suite, it merges. No human in the loop for this lane, and I will defend that below. The results: 708 dependency vulnerabilities closed through the automated program in the core application repositories, with a median time of about 48 hours from PR opened to merged. Steady state, the loop resolves more than 30 findings per week.

First-party application code is the harder lane. Here, AI-generated fixes are raised as proposed pull requests, and AI reviewers work the PR conversation alongside humans, flagging issues, explaining risks, suggesting corrections. AI-authored fixes are merged and running in production code today. Real fixes, in real services, carrying real traffic.

One story from this lane. A routine scan flagged a critical SQL injection risk: string-based query concatenation in a Python service. By the time an engineer opened the finding, an AI-generated fix was already attached, explained, and waiting for review. Fifteen minutes from detection to a fix ready for human eyes. I have been doing infrastructure and security work for over fifteen years. I remember when that class of finding meant a war room and somebody’s lost afternoon.

Stage four: prove it, structurally

Here is the part that makes the previous stage safe, and the part most writing about AI remediation skips entirely.

An AI-generated fix is a hypothesis. Nothing more. So every fix, whether a human wrote it or a model did, passes the same gate chain before it can reach a protected branch: branch protection with required reviews, the full test suite, and a re-scan by the same tools that raised the finding in the first place.

If the vulnerability still appears in the re-scan, the merge is blocked. If tests fail, the merge is blocked. Blocked by the platform, not by a policy document asking people to behave. That distinction is the entire foundation of trust in the system. The pipeline does not care who authored the fix, and that indifference is precisely what makes AI authorship acceptable.

And one deliberate line in the sand: changes touching authentication, data handling, or infrastructure always require a human review, on top of every automated check. We could automate further. We chose not to. Anyone who tells you they have fully automated security remediation is either lying or about to have a very bad quarter.

The results

Lifetime of the program, the pipeline has resolved more than 9,400 security findings. I want to be careful with that number, because big numbers invite skepticism and they should. A large share of it was deliberate burn-down of historical backlog. The honest steady-state claim is the weekly one: 30 or more findings resolved per week, sustained, with no dedicated security headcount.

The changes that mattered most were not the totals:

The triage change. 72 percent of scanner output never reaches a human. Engineers now open a queue that is short, ranked, and real.

The velocity change. Dependency vulnerabilities that used to age for weeks now close at a 48-hour median, automatically.

The trust change. Every fix carries structural proof. The question “did that actually get fixed” has a machine-verified answer.

The cultural change, which surprised me most. Once engineers stopped drowning in false positives, their relationship with security findings changed. A finding in the queue now means something. People fix things faster when they trust that the queue is telling the truth.

What went wrong, because something always does

Early triage tuning was too aggressive and suppressed a finding class it should not have. We caught it, reopened a batch, and retuned. Auto-merge was initially too permissive on one repository and we tightened the gates. The AI still proposes fixes for first-party code that reviewers reject, and that rejection rate is exactly why the human gate on sensitive changes is permanent.

None of these were failures of the concept. All of them were failures of tuning, and the tuning is where the actual expertise lives. The tools are available to everyone. The judgment about where automation ends and humans begin is not in the box.

The lesson

AI does not make security remediation trustworthy. Architecture does.

The model writes the fix faster than any human. The pipeline proves the fix the same way it always should have: tests, re-scans, protected branches, human judgment where the blast radius demands it. Get that order of priorities right, and AI in the SDLC stops being a vendor slide and starts being 277 hours of engineering time you get back.

Would you let an AI-generated fix merge without a re-scan? That question is a useful litmus test for any AI security pitch you hear this year. The right answer has nothing to do with how good the AI is.

If you are building something similar and want to compare notes on triage tuning, validation gates, or where to draw the human review line, my inbox is open.