The Architecture Decision That Looked Perfect. We Paid for It 18 Months Later.

It wasn’t the technology. It was the question we forgot to ask.

The meeting lasted forty-five minutes.

Eight engineers in a room. A staff engineer at the whiteboard. Two competing proposals on the table. A decision that would shape how we built systems for the next three years.

The discussion was good. Technical depth on both sides. Real tradeoffs considered. The team was experienced. Nobody was cutting corners.

We made the wrong decision anyway.

Not because we chose the wrong technology. The technology we chose was fine. It does what we needed it to do. It performs well. The team knows how to use it.

We made the wrong decision because we answered the wrong question.

Eighteen months later, at 2am, with three engineers on a bridge and a system that had been degrading for six hours without anyone understanding why, we finally understood what the right question was.

This is that story.

The Decision

We were building a new notification system.

The existing system was a monolithic component that had grown organically for four years. It handled email, push notifications, and in-app alerts through a single synchronous pipeline. It worked. It was also becoming a bottleneck as notification volume grew and the product team wanted to add new notification types.

The proposal on the table was to extract the notification system into an event-driven architecture using a message queue. Producers would publish notification events. Consumers would handle delivery by channel.

Two message queue options were on the table. RabbitMQ and Kafka.

The discussion that followed was technically sophisticated. Throughput numbers. Persistence guarantees. Consumer group models. Delivery semantics. The team knew the material.

After forty-five minutes, we chose Kafka.

The reasoning was sound. We were already using Kafka for analytics pipelines. The team had existing operational knowledge. Kafka’s throughput ceiling was higher than we’d ever need, but higher ceilings felt like the right kind of problem to have. The persistent log model meant we could replay notifications if something went wrong in the consumer.

The decision looked correct in the room.

The First Eighteen Months

The implementation went well. The new notification system launched on schedule. Producers and consumers worked as designed. Notification volume grew. The system handled it.

For eighteen months, the decision looked correct in production too.

Then we started adding notification types.

The product team wanted transactional notifications — account activity, security alerts, payment confirmations. These notifications had different requirements than marketing notifications. They needed guaranteed delivery. They needed ordering guarantees within an account. They needed the ability to suppress duplicates if a consumer processed the same event twice.

We added the new notification types. We built the consumer logic. We deployed.

The system behaved correctly under normal load.

Under concurrent load — multiple events for the same account arriving simultaneously — it didn’t.

2am, Six Hours In

The incident started with customer support tickets. Users reporting duplicate security alerts. Account activity notifications arriving out of order. Payment confirmations delivered twice.

By the time the on-call engineer was paged, the problem had been happening for three hours across a subset of high-activity accounts.

I joined the bridge at 2am.

What we found over the next six hours was not a bug in the code. The code was doing exactly what it was designed to do.

The problem was that what it was designed to do was wrong for this use case.

Kafka’s consumer model partitions events by key. Within a partition, ordering is guaranteed. Across partitions, it isn’t. Our notification events for a single account were distributed across partitions based on a hashing function that didn’t account for account-level ordering requirements.

The duplicate delivery problem was a different issue. Our consumer logic assumed idempotent processing — that processing the same event twice would produce the same result. For marketing notifications, that assumption held. For transactional notifications, it didn’t. A security alert delivered twice is not the same as a security alert delivered once.

Both problems had the same root cause. We had designed the system for the notification requirements we had eighteen months ago. The requirements had changed. The architecture hadn’t.

And the architecture was now very difficult to change.

The Question We Forgot to Ask

At 4am, after we had stabilized the incident with a temporary fix that limited concurrent processing per account — which tanked our notification throughput by forty percent — one of the senior engineers on the bridge said something I’ve thought about since.

“We never asked what the failure modes were. We asked what the capabilities were.”

That sentence is the entire lesson of this story.

In the architecture meeting eighteen months earlier, we had asked: can Kafka handle our throughput? Yes. Does it give us persistence and replay? Yes. Does the team have operational knowledge? Yes.

We had not asked: what happens when a consumer processes the same event twice? What happens when ordering matters within a subset of events? What does the migration path look like if our requirements change?

Those questions have answers. Kafka has answers to all of them. The answers are not simple — they involve careful partition key design, idempotency tokens, consumer offset management, exactly-once semantics that come with their own tradeoffs — but they’re knowable.

We didn’t know them because we didn’t ask.

We were in a room full of engineers capable of asking those questions. We had forty-five minutes. We had both options documented.

We spent forty-five minutes on capabilities and zero minutes on failure modes.

Why This Happens in Every Architecture Meeting

I have been in a lot of architecture meetings since that one.

The pattern I see consistently is not that teams make bad technical decisions. Most teams in most architecture meetings are technically competent. They know the tools. They can evaluate tradeoffs.

The pattern is that architecture meetings naturally gravitate toward capability discussion.

Can this technology do what we need it to do? How does it perform at scale? What’s the learning curve? What does the team already know?

These are all legitimate questions. They’re also the easy questions. They have answers that are findable in documentation, benchmarks, and team experience.

The hard questions are operational.

What does this look like when it fails? What’s the debugging experience at 2am when something is wrong and you don’t know what? How does it behave when the requirements it was designed for change six months from now? What’s the migration cost if we’re wrong?

These questions are harder to answer because the answers require operational experience with the technology under failure conditions, not just implementation experience under normal conditions.

Teams answer the easy questions well and skip the hard questions because the hard questions don’t have clean answers in the documentation.

The cost of skipping them doesn’t appear for eighteen months.

The Four Questions That Should End Every Architecture Decision

After that incident, I started bringing four questions to every architecture discussion I participate in.

They’re not original to me. They’re the distillation of every postmortem conversation I’ve been in that included the phrase “we should have asked this earlier.”

What does this look like when it fails?

Not if it fails. When. Every system fails. The question is what the failure looks like and whether you’re equipped to handle it.

For message queues: what happens when consumers fall behind? What happens when a message can’t be processed? What happens when the broker itself has a problem? These aren’t hypothetical edge cases. They’re scheduled events. Every system running Kafka long enough will encounter consumer lag, poison pill messages, and broker issues.

If the answer to “what does this look like when it fails” is “I don’t know” or “we’d have to figure it out at the time,” that’s a signal that the operational evaluation is incomplete.

Who debugs this at 3am and what do they have to work with?

This question changes the evaluation frame entirely.

A technology that is elegant to implement and opaque to debug is a different risk profile than a technology that is messier to implement and transparent to debug.

Kafka’s debugging experience — consumer lag monitoring, offset management, partition inspection, replay capability — is rich if you know it. It is deeply confusing if you don’t. If the engineer who will be on-call when something goes wrong is not the engineer who designed the system, that gap matters.

What does this look like in twelve months when the requirements change?

Requirements always change. The question is whether the architecture has a graceful path for the changes that are most likely.

For the notification system, the most likely requirement change was adding notification types with different delivery semantics. That was predictable. We were building a notification system for a product that was actively growing. New notification types were not a surprise eighteen months later. They were scheduled.

The architecture meeting should have included a ten-minute conversation about the most likely requirement changes and whether the proposed architecture had a clean path for them.

What’s the migration cost if we’re wrong?

This question should be asked last, after the other three.

Some architectural decisions are easily reversible. Others are not. The reversibility should be an explicit factor in the decision, especially when uncertainty is high.

Kafka is not easily migrated away from once it’s embedded in a production system at scale. That’s not a criticism of Kafka. It’s a characteristic of the decision that the team making it should understand clearly.

If the answer to “what’s the migration cost if we’re wrong” is “very high,” that’s not a reason to avoid the technology. It’s a reason to be more certain before committing to it.

What the Meeting Should Have Looked Like

The forty-five minute meeting we had was not a bad meeting. The technical discussion was real.

What was missing was the last fifteen minutes.

After the capability discussion was done, someone should have said: okay, now let’s spend fifteen minutes on the failure questions.

What does Kafka look like when consumer lag gets out of control? We should know this before we choose it, not after we’ve built on top of it.

What does the debugging experience look like for an engineer who didn’t design the system? We should think about this now, because the engineer debugging at 2am will often not be the engineer who is in this room today.

What notification requirements are we most likely to add in the next eighteen months? Let’s list them and check whether this architecture handles them cleanly.

If this decision is wrong, what does unwinding it cost? Let’s know that going in.

Fifteen minutes. The incident at 2am eighteen months later took six hours and required a forty-percent throughput reduction as a temporary fix and a three-month architectural remediation project.

The math is not subtle.

The Lesson Is Not About Kafka

I want to be clear about something.

This story is not an argument against Kafka. Kafka is the right choice for many systems, including some notification systems with exactly the requirements we had eighteen months in.

The lesson is about the questions that didn’t get asked, not about the answer that got chosen.

The same story could be told about PostgreSQL versus DynamoDB. About monolith versus microservices. About synchronous versus asynchronous communication. About any architecture decision made by a competent team with good intentions who spent forty-five minutes on capabilities and zero minutes on failure modes.

The technology is not the lesson. The questions are the lesson.

Ask about capabilities. You will get good answers. Also ask about failure modes, debugging experience, requirement evolution, and migration cost. The answers to those questions will sometimes confirm the capability decision. Sometimes they’ll change it. Either way, you’ll walk out of the room with a decision that has been examined from both directions.

The 2am bridge six months later is where architecture decisions get re-examined whether you want them to or not.

Better to do it in the forty-five minute meeting.

The Staff Engineer’s Question

I’ll close with something the staff engineer on that 2am bridge said after we’d stabilized the incident and were writing up the preliminary findings.

Someone asked: how do we prevent this next time?

He said: “Before we decide on a technology, we need to know what happens when it’s wrong. Not if. When.”

That’s the question that wasn’t in the room eighteen months earlier.

It’s in the room now.

It should be in your room too, before the decision is made, not after the incident teaches it.

The architecture decision framework I described the four questions, the failure mode analysis, the operational evaluation is one of the core sections inside *The Senior Engineer Playbook. Eight architecture decisions decoded side by side: the capability discussion, the failure mode analysis, and the operational cost that doesn’t show up until the system is running in production. If you’re making architecture decisions and the operational questions aren’t in the room yet, that’s where I’d start.*