Why Most Teams Don’t Need Kafka Yet

For backend and DevOps engineers evaluating messaging architecture, or anyone who has been paged because of a consumer group rebalance

It was 2:17 AM on a Wednesday when my phone started screaming.

I had been on call for three days. Still half-asleep, squinting at the Datadog alert: order-fulfillment consumer group lag spiking hard. By the time I opened the laptop, the lag was at 180,000 messages and climbing. Notifications and analytics were both timing out. No events received in the last 12 minutes.

One pod had hit a database connection pool issue. Nothing dramatic. A slow query that snowballed because of bad retry logic. That single slow consumer triggered a Kafka rebalance. Everything else in the group paused for almost two minutes while partitions got reshuffled. Even the healthy consumers got dragged into it.

We were processing maybe 500 to 600 messages a minute at peak. Not exactly Netflix numbers.

It was our third Kafka-related 3 AM incident in three months.

I spent the next hour in logs, trying to understand why one flaky pod could take the entire pipeline hostage. The messages were sitting there safely, perfectly ordered, completely useless until the rebalance settled.

A single unhealthy consumer can force a group-wide pause during rebalance, even when every other consumer is healthy.

That was the first time I seriously thought: why are we running Kafka for this?

What Happened Next

Six weeks later we migrated three of our five consumer pipelines to SQS + SNS. Not everything. Two pipelines had real replay requirements and stayed on Kafka. But the notification and analytics pipelines moved.

The next month I was on call again. No pages. The month after that: one page, unrelated to messaging.

That was eight months ago. The SQS pipelines have been boring ever since. Boring is the goal.

The Kafka Reflex

Kafka has become the default answer to any question involving message passing between services. Mention an event-driven architecture in a design review and someone will suggest Kafka before the sentence ends. It carries the weight of seriousness. LinkedIn runs Kafka. Uber runs Kafka. Netflix runs Kafka.

What those companies don’t mention is the platform engineering team behind it.

Kafka is a distributed commit log, built for high-throughput, ordered, replayable event streams at sustained scale. It is operationally complex by design because the problem it solves is genuinely complex. Brokers, partitions, consumer groups, offset management, replication factors, retention policies. Each of these is a knob you will eventually turn, and turning the wrong one costs you an incident. Managed options like Confluent Cloud and MSK reduce the operational burden. They don’t remove the conceptual complexity.

None of this is a criticism of Kafka. It’s a statement about fit.

The Real Trigger for Kafka Is Not Throughput

Most Kafka vs. SQS comparisons frame this as a throughput question. That framing is wrong.

You can run 50,000 messages per second through SQS queues and be fine if you don’t need replay. Conversely, a system processing 200 messages per minute might have a compliance requirement that mandates Kafka the moment an auditor walks in.

The real trigger is two specific requirements arriving together:

Replay means a consumer can rewind to any point in the stream and reprocess from there. This matters for audit logs, event sourcing architectures, GDPR workflows, and systems where a new downstream service needs to bootstrap from historical events. Most teams think they need replay. Most don’t. Ask concretely: name a scenario where you’d actually rewind the consumer. If the answer is vague, you probably don’t need it.

Strict ordering at scale means sequencing across a high-cardinality key space under sustained load. SQS FIFO handles per-entity ordering cleanly using message groups: per user, per order ID, per tenant. What FIFO can’t do is give you total ordering across millions of distinct keys at very high throughput while also supporting replay.

When you need both, with a team that can operate the infrastructure: Kafka. When you need one or neither, you almost certainly don’t.

Compliance is worth naming separately. Even at low volume, an audit requirement mandating immutable, replayable event history justifies Kafka regardless of message rate. Compliance alone is a legitimate trigger. Throughput alone is not.

What You Get With SQS + SNS + DLQ

SNS is the fan-out layer. One message published to an SNS topic is delivered to every subscribed endpoint simultaneously: SQS queues, Lambda functions, HTTP endpoints. The producer publishes once and walks away.

SQS is the durable buffer. Each subscriber gets its own queue and processes at its own pace. If a consumer crashes mid-processing, the message becomes visible again after the visibility timeout expires and another consumer picks it up. Messages can be retained for up to 14 days.

DLQ is the safety net. After a configurable number of failed attempts, the message routes to the Dead Letter Queue instead of being silently dropped.

Kafka couples consumer health through shared group coordination. SNS + SQS isolates failures so one slow consumer cannot stall unrelated workloads.

The key thing we noticed after migrating: a slow or failed consumer in Service A has no effect on Service B. No shared consumer group. No rebalance. This specific failure mode (one slow consumer pausing everything else) disappears in this model. SQS has its own failure modes (backlog explosions, visibility timeout storms, DLQ floods), but that one is gone.

⚠️ Two Things That Will Burn You

Visibility timeout. The default is 30 seconds. If your consumer takes longer, SQS assumes it died and hands the message to someone else. You get silent duplicates. Measure your P99 processing time in production, multiply by six, set that as your timeout. For long-running consumers (report generation, ML inference, payment processing), also use ChangeMessageVisibility as a heartbeat:

import boto3
import threading

sqs = boto3.client('sqs')
def process_with_heartbeat(queue_url, receipt_handle, process_fn):
    """Extends visibility timeout every 60s. VisibilityTimeout must exceed
    heartbeat interval + worst-case processing time."""
    stop_event = threading.Event()
    def heartbeat():
        while not stop_event.wait(timeout=60):
            try:
                sqs.change_message_visibility(
                    QueueUrl=queue_url,
                    ReceiptHandle=receipt_handle,
                    VisibilityTimeout=180
                )
            except Exception as e:
                print(f"Heartbeat failed: {e}")  # Log; don't crash
    threading.Thread(target=heartbeat, daemon=True).start()
    try:
        process_fn()
        sqs.delete_message(QueueUrl=queue_url, ReceiptHandle=receipt_handle)
    finally:
        stop_event.set()

Idempotency. SQS Standard is at-least-once delivery. Duplicates are a fact of life. Check a deduplication key before processing; record it after.

def handle_message(message):
    message_id = message['MessageId']
    if cache.exists(f"processed:{message_id}"):
        return  # Already handled
    process(message)
    cache.set(f"processed:{message_id}", "1", ex=86400)

For hard exactly-once requirements, use SQS FIFO with MessageDeduplicationId. Simpler to misconfigure correctly than Kafka's exactly-once semantics, and covers the vast majority of real-world cases.

Poison Pills and DLQ Patterns

A poison pill is a message your consumer cannot process regardless of retry count. Malformed JSON, an unknown schema, a reference to a deleted record. It hits maxReceiveCount and routes to the DLQ.

Checking the DLQ manually works until you have thousands of messages and no metadata about why each failed. Better: wire a Lambda to the DLQ that enriches each failure with context (original queue, timestamp, receive count, consumer identity) before writing to S3 or DynamoDB. Your DLQ becomes an auditable failure log. For compliance-sensitive systems, it also proves no message was silently dropped.

resource "aws_lambda_event_source_mapping" "dlq_processor" {
  event_source_arn = aws_sqs_queue.service_a_dlq.arn
  function_name    = aws_lambda_function.dlq_enricher.arn
  batch_size       = 1
}

Production-Grade Terraform

The core pattern: SNS topic, subscriber queues with DLQs, queue policies locked to the source topic. Service B follows the identical pattern with service_b substituted throughout.

locals {
  env    = var.environment
  prefix = "${var.project}-${local.env}"
  tags = {
    Project     = var.project
    Environment = local.env
    ManagedBy   = "terraform"
    Team        = var.team
  }
}

resource "aws_kms_key" "sqs" {
  description             = "KMS key for SQS queue encryption"
  deletion_window_in_days = 10
  enable_key_rotation     = true
  tags                    = local.tags
}

resource "aws_sns_topic" "events" {
  name              = "${local.prefix}-events"
  kms_master_key_id = aws_kms_key.sqs.id
  tags              = local.tags
}

resource "aws_sqs_queue" "service_a_dlq" {
  name                      = "${local.prefix}-service-a-dlq"
  message_retention_seconds = 1209600  # 14 days
  kms_master_key_id         = aws_kms_key.sqs.id
  tags                      = local.tags
}

resource "aws_sqs_queue" "service_a" {
  name                       = "${local.prefix}-service-a"
  visibility_timeout_seconds = 300  # Tune down after measuring P99 in production
  message_retention_seconds  = 86400
  kms_master_key_id          = aws_kms_key.sqs.id
  tags                       = local.tags

  redrive_policy = jsonencode({
    deadLetterTargetArn = aws_sqs_queue.service_a_dlq.arn
    maxReceiveCount     = 3
  })
}

resource "aws_sns_topic_subscription" "service_a_sub" {
  topic_arn = aws_sns_topic.events.arn
  protocol  = "sqs"
  endpoint  = aws_sqs_queue.service_a.arn
}

resource "aws_sqs_queue_policy" "service_a_policy" {
  queue_url = aws_sqs_queue.service_a.id

  policy = jsonencode({
    Version = "2012-10-17"
    Statement = [{
      Effect    = "Allow"
      Principal = { Service = "sns.amazonaws.com" }
      Action    = "sqs:SendMessage"
      Resource  = aws_sqs_queue.service_a.arn
      Condition = {
        ArnEquals = { "aws:SourceArn" = aws_sns_topic.events.arn }
      }
    }]
  })
}

# Service B: duplicate the four resources above with "service_b" substituted for "service_a"

visibility_timeout_seconds = 300 not 30. Default is catastrophic for anything doing real work.

kms_master_key_id on both queues and topic. One line. No reason to skip it.

enable_key_rotation = true. AWS rotates the backing key material annually. Your code doesn't change.

FIFO vs Standard

SQS Standard: near-unlimited throughput, at-least-once delivery, best-effort ordering. Use it for notification pipelines, analytics events, background job dispatch. Anything where occasional duplicates are handled by idempotent consumers.

SQS FIFO: exactly-once processing within a 5-minute deduplication window, strict ordering within a message group, 300 messages per second per queue without batching (3,000 with). High-throughput mode pushes this to tens of thousands per second with queue sharding. Use FIFO for payment state transitions per order ID, inventory updates per SKU, user action sequences per session.

The trap: 300 messages per second per queue catches teams who chose FIFO for safety and then hit a traffic spike. Know your peak volume before committing.

Cost Reality: SQS + SNS vs MSK

Costs below are approximations for us-east-1 as of May 2026. Always model your own workload.

Scenario: 1 million messages per day, 3 SQS subscribers, average message size 1KB

• SNS deliveries to SQS are free.
• SQS requests: roughly 180 million per month (receives + deletes across all queues).
• At $0.40 per million: ~$72/month. With buffer for empty receives and DLQ: $75–90/month total for the full SNS + SQS stack.

Same scenario on Amazon MSK (3 x kafka.m5.large brokers)

• Broker cost alone: ~$454/month.
• Plus storage and CloudWatch: $550–700/month for a minimal production setup.

Savings: Roughly $500–600/month (or $6,000–7,000 per year) by using SQS + SNS for this workload.

When Kafka Is the Right Answer

Use Kafka when all of the following are true:

You need event replay and the requirement is concrete. A new analytics service bootstrapping from 90 days of history. A compliance audit requiring point-in-time state reconstruction. An event sourcing architecture where the log is the source of truth.

You need strict ordering at a scale where FIFO queue sharding becomes operationally unwieldy. This bar is higher than most teams realize.

You have a team with the capacity to operate it. Consumer group management, partition strategy, schema registry. Someone owns these. If that person doesn’t exist yet, Kafka creates incidents until they do.

You’re building infrastructure other teams consume as a platform. At that scope, Kafka’s operational investment amortizes across many consumers.

Migrating From Kafka to SQS + SNS

Step 1: Audit replay usage. Check whether any consumer actually resets offsets in production. If replay is unused in practice, migration risk drops significantly. If it is used, don’t migrate that pipeline.

Step 2: Build SQS infrastructure alongside Kafka. Create the SNS topic, queues, and DLQs. Don’t cut over.

Step 3: Dual-publish. Publish to both Kafka and SNS simultaneously. Run for at least one full business cycle. This is your rollback path.

Step 4: Migrate consumers one at a time. Move the lowest-risk consumer first, monitor for a week, repeat. Kafka stays live until you’re confident.

Step 5: Drain and decommission. Cancel the MSK cluster. Savings start immediately.

For 3–5 consumers done conservatively: two to four weeks.

Where This Stack Has Limits

No replay. Once a message is consumed and deleted, it’s gone. SQS retains unprocessed messages up to 14 days but there’s no way to rewind already-consumed ones.

256KB message size limit. Not relevant for most eventing use cases. For large payloads, store in S3 and pass the reference.

Fan-out cost at scale. At extreme fan-out (50 subscribers, 100M messages/day), MSK’s flat infrastructure cost becomes competitive. Model this before committing.

Latency. SQS long-polling waits up to 20 seconds on empty queues; in practice with regular message flow it’s well under a second. If you have a strict sub-second latency SLO on the messaging layer itself, Kafka has a real advantage.

Decision Framework

Start on the left. Move right only when a concrete, named requirement forces you to.

Quick gut check: Can you name a real scenario where you’d rewind a consumer offset in production? If not, you probably don’t need Kafka.

Replay and ordering guarantees, not raw throughput, should drive your messaging architecture decision.

Closing

The MSK invoice is visible on your AWS bill. The real cost is the engineer tuning partition counts on a Friday afternoon, the rebalancing runbook every new hire has to memorize, the postmortem where root cause is “single slow consumer” for the third time that quarter.

SQS + SNS + DLQ doesn’t have that tax. The failure modes are shallow and well-documented. When something goes wrong, you know exactly where to look.

Pick boring. Add Kafka when boring stops working. In most systems, that day never comes.

Going Further

Full implementation wired to a Lambda consumer with S3 event triggers (IAM policies, DLQ config, Lambda event source mapping): Fail-Safe-Queues

How SQS and SNS behave under failure, throughput ceilings, and the production mistakes teams make most often: The AWS Messaging Mistake That Silently Breaks Your Pipelines

• SQS Developer Guide: Visibility Timeout
• Building Event-Driven Architectures on AWS
• AWS MSK Pricing