One Missing Idempotency Key Can Duplicate an Entire Sync Pipeline

Retries are supposed to improve reliability. In distributed systems, they often multiply side effects instead.

TL;DR: Retries are only safe when the operation is idempotent. Without a durable operation ID, a timeout or dropped response can create duplicate records, duplicate workflows, or inconsistent downstream state. The key lesson: design retries as part of the product, not as an afterthought.

A retry is supposed to make a system safer.

In a badly designed sync pipeline, it does the opposite.

It takes a timeout, a dropped response, or a slow upstream and turns it into two customer records, two invoices, two outbound jobs, or one account stuck in a state nobody intended.

The ugly part is that the duplicate often looks valid.

Nothing crashes.

The error rate stays low.

The queue eventually clears.

Then three days later, someone asks why the CRM has two versions of the same customer and nobody can explain which one is correct.

The failure pattern is usually boring

Imagine a sync worker that creates or updates a contact in a CRM.

The worker sends the request.

The CRM processes it.

The network drops the response.

Your worker times out and retries.

If you did not attach a durable operation identifier that both systems respect, the retry is not really a retry.

It is a second mutation.

Now you own a classic production mess:

• The source system says the record synced once.
• The destination system has two records or two side effects.
• Your queue shows a successful retry.
• Support sees a green dashboard while the data is already wrong.

This gets worse when webhooks are involved.

A webhook can be delivered more than once. It can arrive late. It can arrive out of order. If your consumer treats every event as fresh truth, duplication becomes a business bug, not a transport issue.

Why teams miss it

Idempotency failures rarely look dramatic.

They do not present like an outage.

They present like weird support tickets:

“Why does this customer exist twice?”

“Why did this workflow trigger twice?”

“Why did usage get billed twice?”

“Why did the downstream record roll back to an older state?”

The team often investigates the final visible symptom instead of the original ambiguous write.

The root problem was not the duplicate record.

The root problem was that the system could not safely answer this question:

“Have I already performed this business operation?”

The fix is not “retry less”

Retries are not the enemy.

Unsafe retries are.

The real fix is making the write path safe to repeat.

That usually means four things.

First, the caller sends a unique operation identifier.

Not just a request ID for logs. A real idempotency key tied to the business operation.

Second, the receiving system stores that identifier with the mutation.

If you save the key but not the write, you have a new failure mode.

If you perform the write but fail to save the key, you have another one.

Third, repeated requests with the same identifier return the same semantic result.

The second request should not create a second customer.

It should say, effectively:

“I already handled that operation. Here is the result.”

Fourth, downstream consumers need replay protection too.

A safe API write does not help if the queue consumer, webhook processor, or reconciliation job can still duplicate the business action later.

A practical mental model

If a write can be triggered by any of these, it needs idempotency:

• Client retries
• SDK retries
• Queue redelivery
• Webhook redelivery
• Manual replay
• Reconciliation jobs
• Scheduled repair scripts

The real test is not:

“Do we send an idempotency header?”

The real test is:

“Can this exact business operation be attempted again without changing the final outcome?”

If the answer is no, your retry strategy is secretly a duplication strategy.

The production lesson

Distributed systems fail at the seams.

They fail where one service talks to another.

They fail where “success” is ambiguous.

They fail where a timeout hides whether a write happened.

If your retry path is not explicitly designed, your system is not more reliable.

It is just better at repeating mistakes.

Build the operation ID into the contract.

Persist it with the write.

Return the same meaning on duplicate attempts.

Instrument the replay path like it is part of the product.

Because in production, it is.

If you are designing an integration or sync system, I’m always interested in conversations around failure handling and reliability.