AI Didn’t Reduce Our Bugs — Until We Fixed How It Understands the System

Designing a closed-loop workflow for AI in a complex platform

The Problem We Didn’t Expect

We had already started using AI in development.

Developers were using it to:

• analyze the codebase
• understand flows
• speed up implementation

On paper, this should have improved understanding and reduced bugs.

But it didn’t.

We weren’t seeing a meaningful drop in issues.

In some cases, the problems were just changing shape.

That raised an uncomfortable question:

If AI is helping developers understand the system, why aren’t we seeing better outcomes?

What We Started Noticing

When we looked closer, a pattern emerged.

AI was helpful — but only within the limits of the context it had.

• It answered confidently, even when context was incomplete
• It assumed system boundaries were cleaner than they actually were
• It missed cross-service coupling and hidden dependencies
• Different developers ended up with different interpretations of the same system

So while AI felt useful, it was often:

reinforcing partial understanding, not correcting it

The Insight

At some point, it became clear:

AI doesn’t fix misunderstanding. It makes misunderstanding scale faster.

If the system itself is not clearly understood, AI doesn’t resolve that.

It amplifies it.

Before Changing Tools, We Changed the Questions

Instead of looking for better prompts or better models, we stepped back.

Before going further with AI, we asked:

• Do we actually understand how the system works end to end?
• Do we know what is true — and what is just assumed?
• Can we distinguish current behavior from proposed design?
• Where are the real boundaries, and where are they just conceptual?
• Who makes the final decision when AI suggests something?

The answers were inconsistent.

That inconsistency explained the results we were seeing.

The Shift

We stopped asking:

“How do we use AI better?”

And started asking:

“How do we make sure AI is working from the same understanding as everyone else?”

What We Built Instead

We introduced a shared context layer.

Not a code repository.

Not a source of truth.

A place to reconstruct and stabilize understanding before anything changes.

Not All Information Is Equal

The key idea was simple but important:

separate information by level of certainty

We made a clear distinction between:

• current-state → evidence-backed understanding
• target-state → working design (not yet confirmed)
• proposals → ideas under discussion
• decisions → reviewed and approved direction

This avoids a common failure mode:

treating assumptions, designs, and decisions as the same thing

One Critical Guardrail

We also made something explicit:

documentation is not the source of truth Code, infrastructure, runtime behavior, and data still win.

The context layer exists to guide understanding — not replace reality.

Designing for AI, Not Just Humans

We didn’t just document the system.

We defined how AI should interpret it.

• where to start
• how to read different types of documents
• how to handle uncertainty
• when to defer to real system behavior

We also separated usage modes:

• for design and product work → context layer is primary
• for implementation → code and runtime take priority

This prevents AI from confidently acting on incomplete or outdated assumptions.

Standardizing the Starting Point for AI

We also noticed another issue:

Different developers were using AI against the same repository — but getting different levels of understanding.

So instead of standardizing prompts, we standardized context.

We introduced a simple requirement for each repository:

• an AGENTS.md file describing how AI should operate in that repo
• an AI_ENTRYPOINT.md file defining where to start
• an ai-facts.yml file with stable, machine-readable facts
• local documentation explaining repo-specific behavior
• references back to shared platform context

This created a bridge between local implementation and system-level understanding.

The goal wasn’t to control what AI says. It was to control what AI starts from.

What Finally Worked: A Closed Feedback Loop

The real improvement came when we stopped treating AI as a one-time assistant and built a loop.

The workflow became:

• shared context establishes current-state understanding
• product work starts from that context
• senior engineering reviews requirements against system reality
• cross-component analysis produces component-level specs
• architecture review happens at this stage
• component teams implement in their own repositories
• local AI usage is guided by repo-level context and cross-references back to shared context
• after major releases, implementation insights are reflected back into the shared context

This creates a closed loop.

platform understanding informs implementation implementation feeds back into platform understanding

This Loop Is Expensive — and That’s the Point

This process consumes a significant amount of tokens.

But that cost is not accidental.

It is the cost of maintaining alignment across:

• product intent
• architecture decisions
• component-level implementation

In a complex system, token efficiency is not the first problem. Misalignment is.

One More Constraint: Avoid Overthinking at the Edges

Even with a structured system, there is still a local failure mode.

AI tends to overthink when given rich context.

Inside individual repositories, this shows up as:

• exploring unrelated parts of the system
• over-generalizing solutions
• introducing unnecessary abstractions
• drifting away from the actual task

So while the system reduces large-scale misalignment, developers still need to control how AI is used day to day.

In practice:

• keep prompts concise and task-focused
• avoid open-ended analysis when not needed
• use shared context as reference, not something to re-derive every time
• guide AI back when it starts expanding scope

Alignment at the system level doesn’t remove the need for discipline at the interaction level.

What Changed in Practice

This didn’t make everything faster.

But it changed where mistakes happen.

Before:

• misunderstandings surfaced late (integration, production)

After:

• misunderstandings surfaced early (analysis, design)

The result:

• fewer cross-component surprises
• clearer ownership boundaries
• more consistent decisions
• AI outputs that are actually usable in real work

What This Is Not

This is not:

• fully automated development
• AI replacing engineers
• a shortcut to move faster

It adds discipline.

But that discipline trades uncontrolled speed for controlled progress.

Closing Thought

AI is powerful — but only within the limits of how well a system can be understood.

Before adopting AI in a complex environment, it’s worth asking:

• Do we have a shared understanding of the system?
• Do we know what is true, and what is assumed?
• Do we have a way to keep that understanding aligned over time?

If not, AI won’t reduce complexity. It will make it harder to see.

If this feels familiar, you’re probably already experiencing it.

And in that case, the first step isn’t more AI.

It’s making sure everyone — human or not — is working from the same reality.