We Moved to SQS FIFO and Found the Real Failure Modes

How one production migration taught us that ordering is the easy part, and failure modeling is the real work.

👉 GitHub Repository (Java + Node.js examples):** sqs-fifo-reliability-playbook

Photo by Shubham Dhage on Unsplash

We did what many teams do after a few painful retries and race conditions: we moved a critical sync pipeline to SQS FIFO.

On paper, it looked like the reliability chapter was done.

FIFO gave us ordering. FIFO gave us dedup. FIFO gave us confidence.

Then production gave us a different lesson.

The first week after cutover, we had all the right cloud components and still saw weird behavior:

• duplicate side effects
• one customer stream falling behind while everything else looked healthy
• retries that looked “normal” in queue metrics but were quietly amplifying downstream failures

Nothing was fundamentally wrong with FIFO. Our assumptions were wrong.

This is the story of what failed, why it failed, and the design choices that finally made the pipeline boring in production.

The assumption that broke first

Our implicit mental model was simple:

• FIFO means ordered, therefore safe.

What FIFO actually guarantees is narrower:

• ordered delivery per MessageGroupId
• duplicate suppression for the same MessageDeduplicationId in a 5-minute window

What FIFO does not guarantee:

• global ordering across all traffic
• exactly-once business side effects
• automatic recovery from poison-message stalls

That gap between queue-level guarantees and business-level guarantees is where most incidents are born.

Incident 1: “Why are we seeing duplicates if this is FIFO?”

Our first real failure mode came from visibility timeout.

We had treated visibility like a delivery delay knob. It is not. It is a lock duration.

Message lifecycle in practice:

1. Consumer receives message M1.
2. M1 becomes invisible for N seconds.
3. If processing and delete finish before N, done.
4. If not, M1 reappears and gets retried.

When N is too short, the same message reappears while the first attempt is still executing. In a pipeline with side effects, that creates duplicate writes, duplicate API calls, or both.

What fixed it:

• we set VisibilityTimeout = LambdaTimeout + buffer
• we used a 30 to 60 second buffer for normal variance
• we ensured event source mapping constraints were respected (visibility >= function timeout)

Java snippet we now use for long-running handlers to avoid mid-flight reprocessing:

ChangeMessageVisibilityRequest heartbeat = ChangeMessageVisibilityRequest.builder()
    .queueUrl(queueUrl)
    .receiptHandle(receiptHandle)
    .visibilityTimeout(90)
    .build();

sqsClient.changeMessageVisibility(heartbeat);

Heartbeat log snapshot from the demo run:

The latest heartbeat log mirrors the producer style and shows the full mitigation path in order: HEARTBEAT-START, repeated HEARTBEAT-EXTEND, then HEARTBEAT-DONE.

This is the exact behavior we want under long-running work: the same event stays in-flight while visibility is renewed, and it is deleted only after completion.

This removed a surprising amount of “random” retry noise.

Incident 2: “Queue looks fine, so why are updates late?”

The second failure mode was subtler.

Dashboards showed queue depth in a tolerable range. But one tenant kept reporting delayed updates. We initially chased downstream API latency, but the root cause was a single poison message blocking one MessageGroupId.

This is core FIFO behavior: one failed message can stall everything behind it in that group.

The key operational insight was this:

• global queue health can look acceptable while one business stream is completely blocked.

What changed our response time:

1. We began debugging by group ID first, not by aggregate queue stats.
2. We set a conservative maxReceiveCount (for us, 3) and pushed unresolved failures to DLQ quickly.
3. We added deterministic replay tooling so operators could recover a specific stream safely.

The outcome was better isolation: one customer stream could stall without taking down everyone else.

The design decision that unlocked throughput

We originally debated group IDs as an implementation detail. It turned out to be a system design decision.

Bad options we rejected:

• one global MessageGroupId for all messages (correct but serializes everything)
• random group IDs (parallel but can violate ordering where it matters)

What worked for our domain:

• MessageGroupId = customerId#orderId

Producer code became explicit about those IDs:

SendMessageRequest request = SendMessageRequest.builder()
    .queueUrl(queueUrl)
    .messageBody(objectMapper.writeValueAsString(event))
    .messageGroupId(event.getCustomerId() + "#" + event.getOrderId())
    .messageDeduplicationId(event.getOrderId() + "#" + event.getEventTimestamp())
    .build();

sqsClient.sendMessage(request);

Why this held up:

• strict ordering for one order stream
• natural parallelism across unrelated orders
• cleaner incident scope when a single stream fails

Producer log snapshot from the demo run:

This producer run publishes 6 attempts total: 5 unique events plus 1 intentional retry (msg#5) that reuses the same dedup key as msg#2.

The line-level labels (ORDERING-STEP-*, DEDUP-RETRY, PARALLEL-GROUP*) make it easy to validate what each published message is proving.

This was also the point where Lambda concurrency finally mattered. Concurrency only helps when you have enough active groups.

The dedup surprise we almost shipped

Our first dedup idea was MessageDeduplicationId = orderId. It looked elegant, and it was wrong.

Because dedup is only 5 minutes, using a stable entity ID can drop valid rapid updates inside that window.

What we switched to:

• MessageDeduplicationId = orderId#eventTimestamp or upstream eventId

And one important caveat:

• content-based dedup hashes body only, not message attributes
• if uniqueness depends on attributes, set MessageDeduplicationId explicitly

That one change prevented silent data loss during legitimate rapid updates.

Consumer log snapshot from the demo run:

This matching consumer run receives 5 messages (not 6), which is expected: FIFO dedup suppresses the intentional retry (msg#5) while preserving per-group order for the remaining streams.

Exactly-once: the uncomfortable but useful truth

FIFO gave us stronger delivery behavior. It did not give us exactly-once business effects.

We only stabilized outcomes after adding consumer-side controls:

• idempotent writes keyed by event identity
• safe retry behavior for external API calls
• event version awareness for replay and ordering
• Lambda partial batch failure response enabled

Our Java Lambda path now returns only failed message IDs, so successful records are not retried:

List<SQSBatchResponse.BatchItemFailure> failures = new ArrayList<>();

for (SQSEvent.SQSMessage msg : event.getRecords()) {
    try {
        processOne(msg);
    } catch (Exception ex) {
        failures.add(new SQSBatchResponse.BatchItemFailure(msg.getMessageId()));
    }
}

return new SQSBatchResponse(failures);

The retry shape is shown in the repo README diagrams section in sqs-fifo-reliability-playbook. Without partial batch failure, one bad record can cause successful records in the same batch to be retried unnecessarily.

The migration path that avoided a high-risk cutover

What we did not do: a one-step switch from direct fan-out to FIFO-only traffic.

What we did instead:

1. Added idempotency before migration.
2. Added schemaVersion and explicit event identity fields.
3. Documented group and dedup strategies as design artifacts, not code comments.
4. Replayed historical events in staging FIFO.
5. Ran a short dual-write period.
6. Compared side effects, not just queue metrics.
7. Rolled traffic gradually by tenant/entity.

That sequence gave us rollback options and clean evidence at each stage.

What we monitor now (because these actually predict incidents)

We track these continuously:

• ApproximateAgeOfOldestMessage
• ApproximateNumberOfMessagesVisible
• ApproximateNumberOfMessagesNotVisible
• Lambda Errors and Throttles
• DLQ message count

Fast interpretation we use during incidents:

• visible high + not visible low: consumers are not pulling/scaling enough
• not visible high + age rising: handlers are slow or repeatedly failing
• DLQ rising: a permanent failure class likely entered the system

These signals are far more predictive than queue depth alone.

The baseline that made the system boring

After multiple iterations, this became our minimum production baseline:

• FIFO queue with DLQ
• VisibilityTimeout >= function timeout + buffer
• long polling at 20 seconds
• Lambda partial batch failure response enabled
• idempotent consumer path
• alarms on DLQ and queue age

Once this baseline was in place, FIFO became low-drama. Incidents did not vanish, but they became understandable and recoverable.

If you want to see the tradeoff between queue backlog, batch size, and concurrency, use the repo README diagrams section in sqs-fifo-reliability-playbook.

Harness logs snapshot from the demo run:

This is the final confidence check we run before calling the pipeline healthy: 5 unique messages received, per-stream ordering preserved, and the intentional duplicate suppressed.

A concrete event example

Event:

{
    "msg": "msg#2",
    "schemaVersion": 2,
    "eventType": "ORDER_STATUS_UPDATED",
    "eventId": "evt-A100-2",
    "customerId": "C1001",
    "orderId": "A100",
    "status": "SHIPPED",
    "version": 2,
    "eventTimestamp": "2026-08-27T09:10:11.120Z"
}

Keys:

• MessageGroupId = customerId#orderId
• MessageDeduplicationId = orderId#eventTimestamp

Why:

• preserves order where sequence matters
• preserves parallelism where sequence does not
• suppresses trigger retries for the same event

What would break:

• MessageGroupId = customerId (unnecessary serialization)
• MessageDeduplicationId = orderId (drops valid rapid updates)

Operator runbook we keep close during incidents

Queue checks:

1. Verify visibility timeout is greater than or equal to consumer timeout.
2. Inspect visible vs not-visible counts.
3. Inspect age-of-oldest trend.

Consumer checks:

1. Verify partial batch failure is enabled.
2. Compare function errors vs throttles.
3. Check downstream API latency and failure budget.

DLQ checks:

1. Alert threshold should be low (usually greater than 0).
2. Classify transient vs permanent failures.
3. Replay with idempotency guard enabled.

Post-incident hardening:

1. Add the failure as a reproducible test fixture.
2. Add dashboard slices by high-volume group family.
3. Change group/dedup policy only with replay evidence.

Practical code references

Repository: sqs-fifo-reliability-playbook

Closing thought

Moving to FIFO did not remove failure. It changed failure from “chaotic and cross-system” to “localized and diagnosable.” That is a big win, but only if you design for it explicitly.

If your team is in the middle of a similar migration, the most useful architecture question is not “Are we using FIFO?” It is “What exactly happens when one message fails at 2:07 AM, and how fast can we recover one group without risking the rest of the system?”

That question is where reliability actually starts.