Event System Isn’t Decoupled — It’s Quietly Coupled Through Data

“False decoupling” is what happens when services stop calling each other… but still break each other anyway.

The system was healthy.

That’s what made it dangerous.

No errors.

No alerts.

No spike in failures.

The notification service was running. Consuming events. Acknowledging messages. Keeping up with the queue.

From the outside, everything looked fine.

Except users weren’t getting emails.

Order confirmations just… stopped.

Not all of them. Just most.

Which is worse, by the way. Total failure gets noticed immediately. Partial failure lingers. Quietly. Comfortably.

The first report came from a user.

Three days after their order.

We traced it back.

Kafka topic looked normal. Events were flowing. Consumers were alive.

Nothing obviously broken.

And then we saw it.

A field rename.

Seven days earlier.

customer_email → recipient_email

Clean PR.

Reasonable change.

The kind of rename you approve without thinking too hard about it.

The system didn’t crash.

It didn’t even complain.

It just… stopped doing something important.

That’s the moment you realize something uncomfortable:

The system wasn’t decoupled.

It was just quietly coupled in a place you weren’t looking.

The Story We Tell Ourselves About Events

Event-driven systems come with a very appealing narrative.

Services don’t call each other.

No tight coupling.

No cascading failures.

Just events.

Publish what happened.

Let others react.

Everyone moves independently.

It feels clean.

Architecturally elegant.

Almost… mature.

And for a while, it works.

Which is exactly why the problem takes so long to show up.

Because the coupling didn’t disappear.

It moved.

The Thing We Don’t Name

Here’s the part most teams don’t say out loud:

Your services are not coupled through calls. They are coupled through data.

And data has shape.

Structure.

Meaning.

Expectations.

When one service changes that shape…

Every consumer is affected.

Whether you realize it or not.

Let’s name it properly.

Because it deserves a name.

False Decoupling

A system that looks independent at runtime…

…but is tightly coupled through shared schemas.

Once you see it, you can’t unsee it.

Where the System Actually Broke

Before the change, the event looked like this:

{
  "event_id": "ord-291847",
  "type": "order.created",
  "customer_email": "user@example.com",
  "order_total": 183.47
}

After:

{
  "event_id": "ord-291848",
  "type": "order.created",
  "recipient_email": "user@example.com",
  "order_total": 183.47
}

Nothing exploded.

No deserialization error.

No panic.

Just this:

if e.CustomerEmail == "" {
    return nil
}

And suddenly:

No email sent.

No log complaining.

No alert firing.

Because from the system’s perspective…

Nothing was wrong.

That’s the problem with false decoupling.

Failures don’t look like failures.

The First Layer of the Problem: Silence

Go’s JSON parsing is forgiving.

Almost too forgiving.

Unknown fields?

Ignored.

Missing fields?

Zero values.

Which means this:

Your consumer can receive a completely incompatible event… and still succeed.

That’s not resilience.

That’s blindness.

We like to think:

“If something breaks, we’ll know.”

You won’t.

Not always.

Sometimes the system just keeps going.

Slightly worse.

The Second Layer: Drift You Don’t See

Here’s what made this worse.

The publisher and consumer had different structs.

order-service/
  order_created.go

notification-service/
  order_created.go

Same idea.

Different files.

No shared ownership.

They started identical.

Of course they did.

And then they drifted.

Because nothing stopped them.

Every small change was reasonable.

Individually.

Collectively?

They diverged.

Quietly.

This is the part people resist:

The coupling already exists.

You just didn’t acknowledge it.

Sharing a schema package doesn’t introduce coupling.

It makes it visible.

And visible problems are at least debuggable.

The Third Layer: No Early Warning System

The rename shipped.

CI passed.

Tests passed.

Deployment went through.

Why wouldn’t it?

There was nothing checking the contract between services.

That’s the real failure.

Not the rename.

The absence of a moment where the system says:

“Hey… someone else depends on this.”

Without that moment, changes propagate silently.

Until production notices.

Which is always late.

The Fourth Layer: The Bug That Should Make You Nervous

This one is worse.

Because it looks correct.

The order service changes:

"order_total": 183.47

To:

"order_total": 18347

(Cents instead of dollars)

Still valid JSON.

Still a number.

Still parses.

Consumer:

OrderTotal float64

Result?

18347 → 18347.0

Now your system sends:

“Thanks for your $18,347 order.”

No error.

No panic.

Just… wrong.

This is where things stop being annoying…

…and start being dangerous.

A Small Pause

This is the part where people usually say:

“Yeah, but we’d catch that in tests.”

Maybe.

If your tests:

• use real payloads
• cover realistic scenarios
• stay updated

Which, let’s be honest, is optimistic.

The Shift That Changes Everything

At some point, the mental model breaks.

And something replaces it.

Events are not “messages.”

They are not “just data.”

They are:

public APIs with no request-response visibility

And once you see them that way…

Everything changes.

Because now the question becomes:

Would you rename a REST field without versioning?

Of course not.

Then why are you doing it here?

The Contract You Didn’t Write Down

Every event has a contract.

Even if you never wrote it.

• required fields
• optional fields
• data formats
• meaning of values

Consumers rely on that contract.

Implicitly.

And implicit contracts don’t age well.

They drift.

They rot.

They surprise you.

What Actually Works (In Reality, Not Theory)

There’s no single fix.

If there were, everyone would already be using it.

What works is layering constraints.

1. Validate Like You Don’t Trust the Data

Because you shouldn’t.

if e.CustomerEmail == "" {
    return errors.New("missing email")
}

Simple.

Necessary.

Often skipped.

2. Share Schemas (Yes, Really)

People hate this one.

Feels like coupling.

It is.

But again:

The coupling already exists.

You’re choosing between:

• implicit coupling (hidden, dangerous)
• explicit coupling (visible, manageable)

Pick your problem.

3. Contract Tests

This is where things start behaving.

Use real payloads.

Version them.

Test against them.

Because if CI doesn’t fail…

Production eventually will.

4. Version Your Events

Not optional.

order.created.v1
order.created.v2

Breaking changes?

New version.

Parallel publishing.

Gradual migration.

Yes, it’s more work.

No, there’s no shortcut.

The Contradiction Nobody Likes

Event-driven systems are supposed to reduce coupling.

In practice?

They redistribute it.

And distributed coupling is harder to see.

Which makes it harder to fix.

Which is why systems feel “fine”…

Until they don’t.

What This Incident Actually Changed

We added validation.

We added contract tests.

We shared schema definitions.

The fix took hours.

The learning took longer.

Because the real issue wasn’t the rename.

It was the assumption that:

“No direct call means no dependency.”

That assumption is wrong.

The Question That Matters Now

Every time you touch an event:

Who else depends on this?

You won’t always know.

That’s part of the problem.

But asking the question forces something that didn’t exist before:

A pause.

And that pause is where most production bugs quietly die.

Final Thought

The system didn’t fail.

It degraded.

Quietly.

For seven days.

And that’s the kind of failure most architectures aren’t designed to catch.

If your events can change without breaking CI, they can break production without warning.