The Redelivery Trap: Building a Safe Transactional Inbox for Google Pub/Sub

Designing an idempotent, zero-duplication consumer architecture with Golang, GKE, and Postgres.

Idempotency Check

In our previous article, The Transactional Outbox, we built a rock-solid, Postgres-backed staging area. We guaranteed that high-throughput analytical data from BigQuery safely reaches Google Pub/Sub without a single dropped event.

But what about our downstream microservices?

Network flickers… consumer crashes… transient ACK timeouts… What happens next?

The Naive Pattern: “The Blind Trust”

Imagine you spin up a GKE worker or a Cloud Run service that does the following:

• Listen to a Pub/Sub subscription
• Pull a message
• Execute the business logic (e.g., update database state, hit a payment gateway)
• Acknowledge the broker (broker.Acknowledge())

The Failure Mode

The failure mode: “The Blind Trust” pattern

If the network blips after your business logic completes but before the delivery confirmation reaches Google Pub/Sub, the broker will redeliver that exact same message to another pod in your GKE cluster. Your database gets hit again—and this time, it's a duplicate.

If forgetting to process a message means you forgot to charge a client $50k, processing it twice means you just double-billed them. Try explaining to your compliance team that it wasn’t corporate fraud, it was just a “highly available network retry”.

The Solution: The Transactional Inbox

Instead of blindly trusting the network, we make the database the ultimate arbiter of truth. We introduce a simple inbox table with a unique constraint on the message ID.

Instead of a fragile, multi-step process, we execute a single, atomic database transaction:

1. The Gatekeeper (Deduplication): The worker pulls a payload from the broker and attempts to insert the unique message_id into the Postgres inbox table.
2. The Execution (Business Logic): If the insert succeeds, it means this is a brand new message. We execute the business logic in the same transaction, commit, and then tell the broker that the message was successfully processed. If the insert fails (because the ID already exists), we skip the business logic entirely and simply notify the broker we are done to prevent further retries.

The solution: the idempotent transactional inbox

The Technical “Secret Sauce”: ON CONFLICT DO NOTHING

By pushing the deduplication logic down to Postgres, we eliminate race conditions across our entire GKE cluster. We wrap the deduplication and the business state changes in a single transaction.

// 1. Begin a single atomic database transaction
tx, err := db.Begin()
if err != nil {
    return err
}
// ALWAYS defer a rollback to prevent connection leaks if the function returns early.
// If tx.Commit() succeeds later, this becomes a safe no-op.
defer tx.Rollback() 

// 2. Attempt to insert the message ID. 
res, err := tx.Exec(`
    INSERT INTO transactional_inbox (message_id, created_at)
    VALUES ($1, NOW())
    ON CONFLICT (message_id) DO NOTHING;
`, msg.ID) // Don't forget to pass the actual msg.ID variable here!

// 3. CHECK THE ERROR before touching `res`
if err != nil {
    return err 
}

rowsAffected, err := res.RowsAffected()
if err != nil {
    return err // Handle potential driver errors
}

// 4. If rowsAffected is 0, another pod already processed this.
if rowsAffected == 0 {
    // We can explicitly rollback early, or let the defer handle it.
    // We still acknowledge the broker to stop redeliveries.
    broker.Acknowledge(msg.ID) 
    return nil
}

// 5. Execute your core business logic within the SAME transaction
// err = executeBusinessLogic(tx)
// if err != nil {
//     return err // The defer will safely roll this back
// }

// 6. Commit the transaction and safely acknowledge the broker
if err := tx.Commit(); err != nil {
    return err
}

broker.Acknowledge(msg.ID)
return nil

Extra note: Why not use a cache instead?

I often hear developers argue: “Why add database overhead? Just use an in-memory cache like Redis for deduplication!”

Here is the reality:

Caches get evicted; ledgers are forever.

If your Redis cluster restarts, or a key expires too early, your line of defense vanishes. Idempotency isn’t just a fancy backend buzzword — it is the firewall protecting your revenue.

Imagine your service processes finances. When the monthly financial reconciliation report runs, your CFO does not care about transient network partitions or broker availability limits. They only care that the ledger balances. By anchoring your deduplication to the exact same ACID-compliant database that holds your financial state, you guarantee that a message is only ever applied once and the client is billed correctly.