Go Concurrency: How One Server Held 2 Million Goroutines Without Breaking a Sweat

Two million goroutines isn’t the flex you think it is. The scheduler can take it. The thing that fell over was our memory, and that part was on us.

goroutines are the rice: individually almost weightless. The colander is the runtime. What clogs is never the count, it’s what each grain is carrying.

2am, build three, and a memory graph going vertical

An hour into the launch, our memory graph went completely vertical. Connections were past four hundred thousand and still climbing. RAM was right behind them. No plateau, no ceiling, nothing flattening out. We’d bragged about this launch in standup all sprint, and now the on-call channel had gone quiet in the way it does when everyone is staring at the same graph and nobody wants to say it first. Maybe ten minutes before the box tipped.

I assumed a leak, and spent twenty minutes chasing goroutines that wouldn’t exit. Nothing leaked. So I pulled a heap profile, and there was the smoking gun, one allocation dwarfing everything else on the flame graph:

flat  flat%   sum%
   38.10GB 71.42% 71.42%  main.handle  ->  make([]byte, 65536)
    1.40GB  2.62% 74.04%  bufio.NewReader

A single make([]byte, 65536) was eating seventy percent of the heap. The goroutines were fine. They were just fat.

Build one had died weeks earlier from a slow connection leak. Build two choked when a metrics goroutine pegged a core and never let go. By build three we thought we had it nailed, and we were just dying in a brand new way. We kept waiting for the Go scheduler to crack under the connection count. It never did. We just starved the box of RAM. That sentence took us three builds to actually believe. Every time the box died, we went hunting for the clever concurrency bug, the subtle scheduler limit, the thing that would have made a good conference talk. There was no clever bug. There was arithmetic we had refused to do.

What each connection was actually carrying

Our service was a presence layer: sockets that say nothing for hours, then all want a byte in the same second. Nothing happens for ten minutes, then one message lands and a hundred thousand clients need to hear about it at once. Cheap to keep, expensive to wake. That’s backwards from a normal request/response service, and the inversion is what trips people up. Presence layers, live dashboards, websockets, server-sent events, long-poll. Different names, same shape underneath.

Go makes this deceptively easy. You assign a goroutine to every connection and write the handler like it’s the only one alive in the world. The netpoller handles the magic underneath, parking a blocked read on epoll so it doesn’t burn an OS thread while it sits there waiting. A parked connection isn’t holding a thread. The only thing it still pays for is whatever memory you handed it, and we had handed it far too much.

func handle(conn net.Conn) {
    readBuf  := make([]byte, 64*1024) // grabbed up front, held for the life of the conn
    writeBuf := make([]byte, 64*1024) // mostly sitting idle
}

Two 64KB buffers. That’s 128KB sitting resident per connection, allocated whether the socket ever says a word. On a 64GB box that’s a wall around 450,000 connections, which is give or take exactly where we fell over. Nobody had bothered to multiply 128KB by half a million sockets. The profiler did it for us at 2am, in red.

The fix was rent control

A parked goroutine costs its stack, about 2KB to start, plus anything you allocated and never gave back. So we made idle connections hold almost nothing, and rented the big buffers out only while bytes were actually moving.

var bufPool = sync.Pool{New: func() any { b := make([]byte, 32*1024); return &b }}

func handle(conn net.Conn) {
    buf := make([]byte, 4*1024) // 4KB is plenty for our ping/pong and control frames
    for {
        conn.SetReadDeadline(time.Now().Add(90 * time.Second))
        n, err := conn.Read(buf)
        if err != nil { return }
        out := bufPool.Get().(*[]byte)
        m := encodeInto(*out, buf[:n])
        conn.SetWriteDeadline(time.Now().Add(10 * time.Second))
        if _, werr := conn.Write((*out)[:m]); werr != nil { bufPool.Put(out); return }
        bufPool.Put(out)
    }
}

We shrank the pooled buffer to 32KB because our max frame size never came anywhere close to 64KB, and the resident per-connection buffer to 4KB because that holds a ping comfortably. The big buffer now only has to exist during an actual read and write. That took the baseline cost per connection from 128KB down to about 6KB, stack included. A box that used to choke at 450,000 connections was suddenly coasting, and two million of them fit into roughly 12GB of RAM. We shipped it on a Friday afternoon, which I would not recommend to anyone, and watched the memory graph flatten into a boring horizontal line for the first time in three builds. Boring is the most beautiful thing a production graph can be. I almost screenshotted it.

The fix had nothing to do with concurrency. We were paying 128 kilobytes of rent on sockets that sat silent all day.

We pushed the fix, and the CPUs started screaming

Memory plummeted. We packed four times the connections onto the box. And then the garbage collector woke up.

The memory crisis became a CPU crisis overnight: same box, same load, suddenly pinned. A go tool pprof cpu profile showed the time wasn't in our code at all. It was in runtime.gcBgMarkWorker, the background sweeper grinding through a live set that had quadrupled the instant we packed more connections in.

The live set the GC walks every cycle was now four times bigger, and it started burning around 30 percent of a core on a service that mostly sits still. It got worse: sync.Pool is not free from the collector's point of view either. The GC drains the pool on every cycle, so the buffers we were so proud of pooling kept getting reclaimed and reallocated under steady load, which is its own small tax.

debug.SetGCPercent(200)        // fewer cycles, more memory used
debug.SetMemoryLimit(48 << 30) // GOMEMLIMIT: a SOFT 48GB target the runtime works to stay under

GOGC=200 on its own would have been reckless, since it lets the heap grow toward 3x live and re-invites the exact OOM we'd just escaped. GOMEMLIMIT is the real lever, a soft ceiling the runtime works to stay beneath. After both, GC CPU dropped into single digits and stayed there.

If you take one number from this whole post, make it that one. At this scale, GC cost tracks your live object count, not your allocation rate. The cheapest way to cut it is almost always to keep fewer live things around, not to tune the collector harder, and shrinking the per-connection footprint quietly did both at once.

2 million sockets, 64GB of RAM

That was the whole fix. No event loop, no rewrite, just a smaller footprint per goroutine and one GC knob. Past two million connections, the thing that breaks is no longer Go. It is the operating system, and the kernel is a landlord too. It wants its cut.

• TCP buffers. We dropped tcp_rmem and tcp_wmem to a 4KB floor. The kernel reserves a send and receive buffer per socket, and at two million sockets the defaults alone can quietly claim tens of gigabytes you never see in your own heap.
• File descriptors. Two million inbound sockets means two million open fds, and the default ulimit laughs at that number, so we raised it to three million.
• TLS. We terminated TLS at the load balancer, and this one is not a footnote. A TLS session historically costs around 30KB of state. Across two million connections that is 60GB, which is the entire box. Keeping crypto state off this server is half the reason the whole architecture survives. Every layer wants rent: the Go heap, the kernel’s socket buffers, the TLS session state, the descriptor table. Hold two million of anything and a number that looked like a rounding error suddenly weighs as much as your server.

How to kill this setup anyway

Holding two million idle sockets is the easy part. The nightmare starts when they all wake up in the same millisecond: a thundering herd with a GC bill attached, where a broadcast to every connection at once can stall you cold. Stagger the fan out. We learned that one the loud way, when a single config push told all two million clients to reconnect at the same moment, and the reconnect storm took the box down harder than the original memory bug ever had.

The quieter killer is leaks, and on this kind of server a leak is never just a leak. A deadlocked goroutine is a permanent memory leak that holds its stack and its buffers hostage until the box OOMs. Leave one parked on a channel nobody closes, or a write with no deadline, and it never dies. You don’t crash. You stack up zombies for a week and fall over the following Tuesday with no obvious cause. Put a deadline on every blocking call, and go stare at your goroutine count in prod before you trust a single number in this post.

Enjoyed the read? Let’s stay connected!

• 🚀 Follow The Speed Engineer for more Rust, Go and high-performance engineering stories.
• 💡 Like this article? Follow for daily speed-engineering benchmarks and tactics.
• ⚡ Stay ahead in Rust and Go — follow for a fresh article every morning & night.

Your support means the world and helps me create more content you’ll love. ❤️