Idempotency, Deduplication, and Exactly-Once in Distributed Pipelines

Idempotency, deduplication, and exactly-once are often mentioned together in distributed systems, but they represent very different guarantees:

• Idempotency is operation-level safety
• Deduplication is data-level safety
• Exactly-once is end-to-end safety

𝗜𝗱𝗲𝗺𝗽𝗼𝘁𝗲𝗻𝗰𝘆 It performs the same operation multiple times produces the exact same result as performing it once. For example, charging a credit card with an idempotent API call ensures the customer is billed once, even if the request is retried.

𝗗𝗲𝗱𝘂𝗽𝗹𝗶𝗰𝗮𝘁𝗶𝗼𝗻 It detects and discards duplicate requests so the operation executes exactly once. For example, dropping repeated webhook events from the same source so system doesn’t process the same event multiple times.

𝗘𝘅𝗮𝗰𝘁𝗹𝘆 𝗢𝗻𝗰𝗲 It ensures that each operation affects the system only one time, even if the request is retried or duplicated. This is the ultimate guarantee, but in distributed systems, it is mostly an illusion.

Most message queues (like Kafka, RabbitMQ, SQS) provide at-least-once delivery. This means a message might be delivered more than once, which is by design.

𝗧𝗵𝗲 𝘁𝗵𝗿𝗲𝗲 𝗿𝗲𝗮𝗹𝗶𝘁𝗶𝗲𝘀: At-most-once — message may be lost, but never duplicated At-least-once — message always delivered, may be duplicated Exactly-once — a marketing lie with very good PR 😊

𝗔 𝗿𝗲𝗮𝗹-𝘄𝗼𝗿𝗹𝗱 𝗲𝘅𝗮𝗺𝗽𝗹𝗲:

• worker pulls a payment request from Kafka.
• worker charges the customer’s credit card.
• worker crashes before committing the Kafka offset.
• kafka redelivers the same payment request.
• the next worker charges the card again, resulting in a duplicate charge.

This happens by design. Trying to implement exactly-once at the system level is always complex, expensive, and often unnecessary. Instead, ensuring idempotent operations and deduplication is simpler, reliable, and practical and the trade-off is consumers must handle duplicates.

At-least-once + idempotent consumers = effective exactly-once, and this is the production standard.