SQS vs Kafka — Deep Dive for L6/L7 System Design Interviews

The Mental Model First

SQS is a message queue. Kafka is a distributed commit log.

This one sentence drives almost every other difference. SQS is optimized for task distribution — messages are consumed and deleted. Kafka is optimized for event streaming — messages are retained and replayable. When you internalize this, the rest follows naturally.

Architecture Internals

SQS

• Fully managed, serverless. You don’t own brokers.
• Two flavors: Standard (at-least-once, best-effort ordering) and FIFO (exactly-once within a message group, strict ordering, 3,000 msg/s with batching).
• Messages are deleted after consumption — there’s no concept of offset or replay.
• Visibility timeout is the core delivery mechanism: a message becomes invisible after a consumer picks it up; if not deleted within the timeout, it reappears for another consumer.
• Dead Letter Queue (DLQ) is native — after N failed receives, message routes to DLQ automatically.

Kafka

• You own (or manage via Confluent/MSK) the brokers, ZooKeeper/KRaft, and partition topology.
• Core abstraction: topics → partitions → offsets. A partition is an ordered, immutable append-only log.
• Messages are retained based on time or size, not consumption. Consumers hold their own offset — the broker doesn’t care if you’ve read a message.
• Consumer Groups: each group gets a full copy of the stream; within a group, each partition is owned by exactly one consumer.
• Replication factor (typically 3) provides durability. ISR (In-Sync Replicas) list is what determines commit durability.

The 10 Axes That Matter in Interviews

1. Message Ordering

Deep dive point: In Kafka, if you need all events for a given entity in order, you partition by that entity’s key. This means hot partitions are a real risk — if one user generates 10x traffic, that partition becomes a bottleneck. Solutions: salting keys, separate topics for high-volume entities, or partition-aware producers.

2. Consumer Model

• SQS: Push-pull, competing consumers. Only one consumer processes a given message (it’s hidden from others via visibility timeout). This is the classic worker-pool model.
• Kafka: Pull-based. Multiple independent consumer groups can each read the full stream at their own pace. A new analytics team can subscribe to your payments topic without affecting your existing payment processor.

Interviewer probe: “You have a payment event. Your fraud service, notification service, and analytics pipeline all need it. How do you design this?”

• With SQS: You fan out via SNS → multiple SQS queues (one per consumer). Each queue is separate infrastructure.
• With Kafka: Single topic, three consumer groups. Zero fan-out infrastructure overhead. This is where Kafka wins cleanly.

3. Throughput & Latency

Deep dive: Kafka’s throughput comes from sequential disk I/O. Kafka writes to disk, but it’s append-only to a segment file, which is predictable and fast (often faster than random memory access). It also uses sendfile() syscall for zero-copy transfer from disk to network.

4. Delivery Semantics

This is a favorite deep-dive area for senior interviews.

Kafka Exactly-Once internals (interviewers love this):

• Idempotent producer: Each producer gets a PID (Producer ID) and a sequence number. Broker deduplicates retries.
• Transactions: Producer wraps a read-process-write (consume → transform → produce) in a transaction. Uses a two-phase commit via a Transaction Coordinator. Consumer reads only committed offsets if isolation.level=read_committed.
• Cost: ~20% throughput reduction for transactional writes.

5. Replay / Reprocessing

This is Kafka’s killer feature that SQS cannot match.

SQS: Once consumed and deleted, the message is gone. You cannot replay.

Kafka: Reset consumer group offset to any point — beginning, a specific timestamp, or a specific offset. This enables:

• Bug fix and replay: Deploy a fixed consumer, reset offset, reprocess.
• New service bootstrap: New service joins, reads full history to build state.
• Event sourcing: Kafka as the source of truth.

Interviewer probe: “Your fraud model had a bug for 6 hours and missed 50,000 transactions. How do you recover?”

• With SQS: You need a separate audit log (S3, DB) to replay. The queue is empty.
• With Kafka: Spin up a new consumer group, reset to T-6h, replay. Done.

6. Retention

• SQS: Max 14 days, messages deleted after consumption.
• Kafka: Configurable per topic — time-based (default 7 days), size-based, or infinite with log.retention.ms=-1 (bounded by disk). With tiered storage (Confluent, MSK), effectively unlimited.

7. Backpressure & Flow Control

• SQS: Natural backpressure — messages just pile up in the queue. Consumers scale out (with Lambda or ECS autoscaling on queue depth via CloudWatch). Queue depth is a first-class metric. ApproximateNumberOfMessages is your autoscaling signal.
• Kafka: Consumer lag (offset difference between latest and committed) is your backpressure signal. But Kafka doesn’t slow down producers — it’s the consumer’s problem to keep up. If a consumer group falls too far behind and retention expires, you get offset out of range errors — messages are permanently lost for that consumer. This requires active lag monitoring (Burrow, Confluent Control Center).

Deep dive trap: “What happens if your Kafka consumer can’t keep up with the producer?” Kafka won’t push back on the producer (unlike some queue systems). The consumer simply falls further behind. If retention is 7 days and your consumer is down for 8 days, those offsets are gone. You need alerting on consumer lag, not just errors.

8. Operational Complexity

Interviewer probe on Schema Registry: “How do you handle schema evolution in Kafka?”

• Use Avro/Protobuf with Confluent Schema Registry.
• Enforce backward compatibility (new schema can read old data) for consumers.
• Enforce forward compatibility (old schema can read new data) if you have slow-rolling consumers.
• Full compatibility = both = safest for production.

9. Scaling Model

SQS: Scale consumers horizontally. No partition limit to worry about. Each consumer independently polls and processes. Lambda integration is native.

Kafka: Parallelism is bounded by partition count. If a topic has 12 partitions, max parallelism = 12 consumers in a group. You cannot add a 13th consumer that does useful work — it sits idle.

• Increasing partitions: You can increase but not decrease. Rebalancing is needed.
• Partition count planning: Rule of thumb — set partitions to max expected consumer count * 2-3 upfront. Changing later is painful.

10. Cost Model

SQS: Pay per API call (request). High-throughput systems with small messages can get expensive because each receive/delete is a request.

Kafka (MSK): Pay for broker instances (always on). Expensive at low throughput, very cost-efficient at high throughput due to batching.

Break-even: Roughly above ~50–100 million messages/day, Kafka starts being cheaper than SQS at scale. Below that, SQS wins on cost + simplicity.

The Decision Framework for Interviews

When an interviewer asks you to choose, use this framework out loud:

1. Do consumers need to REPLAY events?          → Kafka
2. Do multiple independent services consume?     → Kafka (fan-out is free)
3. Is this a simple task queue (jobs/workers)?   → SQS
4. Do you need Lambda/serverless consumption?    → SQS (native trigger)
5. Is ordering per-entity required?              → Kafka (partition by entity key)
6. Is exactly-once critical (e.g., payments)?   → SQS FIFO or Kafka transactions
7. Are you AWS-only with low ops budget?         → SQS
8. Is throughput > 10K msg/s sustained?          → Kafka

High-Signal Scenarios Interviewers Use

Scenario 1 — Ride-sharing location updates (Uber/Lyft scale)

• Millions of GPS pings/second → Kafka (throughput, fan-out to ETA, surge pricing, mapping services)
• Partition key = driver_id (ordered location stream per driver)
• Interviewer will probe: hot partitions for drivers in dense areas → salting or separate high-density topic

Scenario 2 — E-commerce order pipeline

• Order placed → inventory, payment, notification, analytics all need to react
• SQS: SNS fan-out to 4 queues (works but operationally messy at scale)
• Kafka: One orders topic, 4 consumer groups, zero fan-out infra
• Probe: What if the notification service is slow? Does it affect inventory? No — independent consumer groups.

Scenario 3 — Financial audit trail

• Immutability + replay required → Kafka with long retention or compacted topics
• Probe: “Can a consumer modify a Kafka message?” No. The log is immutable. Compacted topics keep only the latest value per key (useful for CDC/changelog).

Scenario 4 — Async job processing (image resizing, email sending)

• Classic worker queue pattern → SQS wins. Simple, no fan-out needed, Lambda integration, DLQ is built-in.
• Probe: “Why not Kafka here?” Operational overhead for no benefit. You don’t need replay, you don’t need multiple consumers of the same job.

The Subtle Things That Signal Seniority

1. Kafka consumer group rebalancing — The “stop the world” problem. When a consumer joins or leaves, all partitions are reassigned. With eager rebalancing (old default), all consumers stop during reassignment. With cooperative sticky rebalancing (Kafka 2.4+), only affected partitions move. Know this when discussing high-availability consumers.
2. Log compaction vs retention — Retention deletes old segments by time/size. Compaction keeps the latest record per key indefinitely. Use case: CDC (Change Data Capture) where you want current state of every database row.
3. SQS visibility timeout as a design lever — Set it to your P99 processing time + buffer. Too short → duplicate processing. Too long → slow failure recovery. For Lambda, max execution time caps this at 15 min.
4. Kafka as a database — With compaction + Kafka Streams or ksqlDB, Kafka becomes a stateful processing engine. You can do joins, aggregations, windowing on streams. This is event streaming, not just messaging.
5. MSK vs self-managed vs Confluent — MSK gives you managed Kafka but no Schema Registry (you add it). Confluent Cloud gives you the full ecosystem. Self-managed gives full control but ops burden. At a FAANG interview, acknowledging MSK as the pragmatic AWS choice shows operational maturity.

The One-Liner Closer

When wrapping up your recommendation in an interview, land on something like:

“I’d use Kafka here because we have multiple downstream consumers with different scaling characteristics, and the ability to replay events for new services or recovery scenarios is worth the operational overhead. If we were early stage or this was a simple job queue, I’d default to SQS and move fast.”