Designing a Realtime Chat System for Massive Scale (100M Users) — Including Live Streaming Chat

Building a realtime chat system for millions of users is not about adding WebSocket and calling it “real-time.”

At massive scale — 100 million registered users and 10 million active users — the real challenges are not message sending. They are:

• Consistency
• Ordering
• Failure handling
• Regional distribution
• Recovery
• Fanout efficiency

And when you add live streaming chat into the mix, the architecture must evolve further.

This article walks through a practical, production-grade approach to building such a system.

Functional Requirements

Let’s define what the system must support.

• 1:1 chats
• Small & medium groups
• Live streaming chat rooms
• Same user may connect from mobile + web simultaneously.
• Cursor-based retrieval.
• No message loss even after crash.
• Multi-region deployment.
• Regional failover.
• 10M active WebSocket connections.
• Hundreds of thousands of messages per second.

Non-Functional Requirements

These matter more than features at scale:

• Horizontal scalability
• Exactly-once persistence
• At-least-once delivery
• Observability
• Fault tolerance
• Cost efficiency (network dominates at scale)

High-Level Architecture

At massive scale, responsibilities must be separated clearly.

Core Components

• EKS WebSocket Service (connection handling)
• EKS Message Service (ingestion)
• Kinesis (event backbone)
• DynamoDB Global Tables (durable store)
• Redis (regional connection registry)
• Route 53 (regional health routing)

Connection Management at 10M Active Users

Each WebSocket connection is handled by EKS pods.

We store ephemeral connection state in Redis.

Redis Structures

user:{userId}:connections → SET(connectionIds)
connection:{connectionId} → HASH(serverId, region, userId) + TTL

TTL is used for:

• Crash recovery
• Zombie cleanup
• Self-healing state

Connections are distributed randomly across pods to prevent hotspots.

We do NOT colocate users of the same conversation on the same pod.

Log-First Message Ingestion

We do not write directly to DynamoDB.

Instead:

Client → Message Service → Kinesis

Kinesis provides:

• Ordered ingestion per shard
• Replay capability
• Backpressure handling

Exactly-Once Persistence

Kinesis consumer writes to DynamoDB using conditional writes:

ConditionExpression: attribute_not_exists(messageId)

This guarantees:

• No duplicate storage
• Idempotent retries
• Stable ordering (using ULID)

Persistence is the source of truth.

Realtime Delivery Model

After successful DynamoDB write:

1️⃣ Same-region WebSocket delivery 2️⃣ Cross-region via DynamoDB Global Table replication

Delivery is at-least-once. Client performs deduplication using messageId.

Standard Group Chat Fanout

For normal groups (3–5000 users):

Flow:

1. Get conversation members from durable store.
2. For each user, check Redis for active connections.
3. Group connections by serverId.
4. Publish one message per serverId.
5. Each WebSocket pod sends locally.

This scales perfectly for small and medium groups.

The Live Streaming Chat Problem

Live streaming changes everything.

You may have:

• 100K–500K concurrent viewers
• All in the same chat room
• Messages flowing rapidly

Naively resolving:

For each user in group:
    Lookup connection

does not scale.

Scalable Fanout Strategy for Live Streaming

The optimization is simple:

Instead of resolving per-user at message time, we distribute by server ownership.

Because connections are already distributed across pods:

• Each pod owns a subset of connections.
• Fanout should be distributed by pod.

So instead of:

Send to 500K users

We:

1️⃣ Determine which pods have active connections for this conversation. 2️⃣ Publish one message per pod. 3️⃣ Each pod delivers locally to its users.

Now fanout cost becomes:

OrderOf(number_of_pods)

Instead of:

OrderOf(number_of_users)

This is the key to scaling live chat.

Avoiding Over-Engineering

Not every chat system needs ultra-optimized routing.

Most real-world systems are dominated by:

• 1:1 chat
• Small groups

Design for realistic workloads first.

Only optimize live-stream fanout when:

• Group size becomes large.
• Broadcast frequency becomes high.
• Metrics justify the complexity.

Failure & Recovery Model

If a client crashes:

• It reconnects.
• Sends lastSeenMessageId.
• Retrieves missed messages via cursor-based pagination.

Realtime delivery is optimization. Durable storage ensures correctness.

What Really Breaks at 10M Scale?

Not Redis. Not DynamoDB. Not Kinesis.

The real bottlenecks are:

• CPU (TLS encryption during fanout)
• Network egress
• Reconnect storms during regional failures

Observability and autoscaling matter more than micro-optimizations.

Live Streaming at Ultra Scale

If your product includes massive live rooms:

Instead of dynamically resolving 10M users per message,

you can maintain server-level routing metadata per conversation dynamically — so fanout becomes server-based rather than user-based.

This avoids resolving millions of connections during broadcast.

Final Thoughts

Building a chat system for 100 million users is not about WebSocket libraries.

It’s about:

• Separating ingestion from delivery
• Separating durability from transport
• Embracing at-least-once delivery
• Designing for failure
• Scaling horizontally
• Optimizing fanout distribution

Realtime systems are not about perfect delivery.

They are about eventual correctness with minimal latency.

And at massive scale, distribution is everything.