The Go Scheduler Didn’t Work How I Thought — Until I Read the Source Code

Most explanations stop at ‘goroutines are lightweight threads.’ Here’s what actually happens inside GOMAXPROCS, the work-stealing queue, and why that matters when you’re debugging a CPU spike.

CPU usage was at 94%. Goroutine count looked normal — around 340, well within what the service had handled before. Nothing was blocked. No lock contention showing in the mutex profile. The latency was inconsistent in a way that suggested scheduler behavior, but I had no framework for thinking about what the scheduler was actually doing.

I spent two days reading documentation and blog posts before I gave up and read the source code instead.

The documentation describes goroutines as lightweight threads multiplexed onto OS threads. That is technically accurate. It is also almost entirely useless for understanding what was happening in our service. The source code told a different story — one about queues, preemption, work stealing, and scheduling decisions that are invisible unless you know where to look.

What follows is what I learned. Not a comprehensive tour of the runtime. The parts that changed how I think about concurrency in Go, and that would have shortened that two-day debugging session considerably.

The Mental Model That Almost Everyone Starts With

Goroutines are lightweight. You can run thousands of them. The Go runtime handles the scheduling. This is the summary that most introductions provide, and it is enough to write concurrent Go code that works in most cases.

The mental model that follows from this summary: more goroutines means more parallelism, GOMAXPROCS determines how many things run at once, and if your service is CPU-bound, you want GOMAXPROCS to match your core count.

None of this is wrong exactly. All of it is incomplete in ways that matter when something goes wrong.

The piece that is missing: goroutines are not scheduled uniformly. The scheduler makes decisions based on local run queues, a global run queue, OS thread availability, and a work-stealing mechanism that moves goroutines between threads. Those decisions are not random — they follow rules — but the rules are runtime-internal and not directly observable from your code.

When the scheduler’s behavior diverges from your intuition, the divergence is usually traceable to one of these mechanisms. Understanding them does not require reading all of runtime/proc.go, but it does require going past the "lightweight threads" abstraction.

The M, P, G Model (What the Scheduler Actually Manages)

The Go runtime scheduler manages three types of entities. This is documented, but the documentation is sparse enough that the implications are easy to miss.

G — Goroutine. The goroutine itself. Contains the stack, the program counter, scheduling state. There can be millions of these.

M — Machine. An OS thread. The entity that actually executes code on CPU. GOMAXPROCS does not limit the number of M's—it limits the number of M's that can run Go code simultaneously. M's can exist and be blocked in syscalls while other M's run Go code.

P — Processor. A logical processor. This is the entity GOMAXPROCS controls. Each P has a local run queue—a ring buffer holding goroutines ready to execute. A goroutine must be assigned to a P to run. A P must be assigned to an M to execute. The relationship: G runs on P, P runs on M.

             ┌──────────────────────────────────────┐
             │           Go Runtime                 │
             │                                      │
             │   ┌─────────┐    ┌─────────┐         │
             │   │    P0   │    │    P1   │         │
             │   │ runq:   │    │ runq:   │         │
             │   │ [G1,G2] │    │ [G3]    │         │
             │   └────┬────┘    └────┬────┘         │
             │        │              │              │
             │   ┌────▼────┐    ┌────▼────┐         │
             │   │   M0    │    │   M1    │         │
             │   │(OS thd) │    │(OS thd) │         │
             │   └─────────┘    └─────────┘         │
             │                                      │
             │   Global run queue: [G4, G5, G6]     │
             └──────────────────────────────────────┘

When you call go func(), the new goroutine is placed on the current P's local run queue. If the local queue is full (it holds at most 256 goroutines), half the goroutines are moved to the global run queue. This is the first place things diverge from the simple mental model: goroutines do not go into a single shared queue. They go into a P's local queue, which is only accessible to the M running that P.

Work Stealing: Why Your Goroutines Don’t Run Where You Expect

When a P’s local run queue is empty, it does not sit idle. It steals work.

The stealing algorithm (in runtime/proc.go, function findRunnable): the idle P checks the global run queue, then checks netpoll (for goroutines waiting on network I/O), then steals from another random P's local queue.

Stealing takes half the victim P’s queue. If P1 has 14 goroutines waiting and P0 is idle, P0 steals 7 of them and starts executing. The goroutines move from P1’s context to P0’s context.

This is efficient. It keeps CPUs busy. It is also why goroutine execution order is unpredictable even when you think the queue is ordered, and why a goroutine spawned in one goroutine’s context may execute on a completely different OS thread.

For most code, this does not matter. For code that holds per-thread state, or code that assumes goroutines run close to where they were spawned, or code debugging why certain goroutines are experiencing high latency, it matters considerably.

The practical implication: you cannot reason about scheduling locality from your Go code. If goroutine A creates goroutine B, B may run on the same P immediately (if P’s queue is processed in order), or may be stolen by another P before A even finishes, or may sit in the global run queue for several scheduling cycles. The order depends on the current state of all P queues, which changes at the microsecond level.

GOMAXPROCS: What It Controls and What It Doesn’t

GOMAXPROCS sets the number of P's—logical processors—available to run Go code simultaneously. The default since Go 1.5 is the number of logical CPUs on the machine.

The common assumption: GOMAXPROCS equals the maximum concurrency. If you have 8 cores and GOMAXPROCS is 8, you have 8 goroutines running at once.

This is true for CPU-bound goroutines. For I/O-bound goroutines, it is wrong in an important way.

When a goroutine makes a blocking syscall — reading from a file, waiting on a network connection — the M running that goroutine is handed off to handle the syscall. A new M is obtained (from a pool or created) to continue running the P’s work queue. The goroutine in the syscall is detached from its P. The P continues executing other goroutines.

The result: the number of M’s (OS threads) in existence can exceed GOMAXPROCS significantly. GOMAXPROCS limits how many M’s are actively running Go code. It does not limit how many M’s exist in total.

In a service with heavy I/O, you might have GOMAXPROCS=8 and 47 OS threads. 8 are running Go code. The rest are blocked in syscalls. The Go runtime manages this invisibly — creating and parking M’s as needed. It is not a problem in the usual case. It becomes relevant when you are looking at OS-level thread counts and finding numbers that don’t make sense from GOMAXPROCS alone.

Preemption: Why Goroutines Don’t Run Forever

A goroutine that is running CPU-bound work could theoretically hold its M indefinitely. The early Go scheduler (before 1.14) had this problem — goroutines could starve others in compute-heavy loops.

Go 1.14 introduced asynchronous preemption. The runtime sends signals to OS threads running goroutines, forcing them to yield at safe points. Goroutines can now be interrupted even in tight loops with no function calls.

This is mostly invisible in well-behaved code. It becomes relevant in two situations.

First: profiling. CPU profiles are collected via signals. If a goroutine is preempted, the profiler captures the signal at the preemption point. This is usually fine. For very tight loops or very short-lived goroutines, the profiling signal may not land frequently enough to produce accurate samples.

Second: garbage collection. The GC requires all goroutines to be at safe points for certain phases (STW — stop the world). Asynchronous preemption is what makes this work correctly even for goroutines in tight loops. The latency cost of STW phases is directly affected by how quickly goroutines can be brought to safe points, which is affected by preemption responsiveness.

For most service code, you do not need to think about this. For understanding why GC pause times vary, or why CPU profiles show surprising call stacks, the preemption mechanism is the relevant context.

What We Were Actually Seeing

Back to the original incident: 94% CPU, normal goroutine count, inconsistent latency.

The runtime trace told the story that pprof alone could not. Running a trace:

import "runtime/trace"

// Start trace
f, _ := os.Create("trace.out")
trace.Start(f)
defer trace.Stop()

// ... service handles requests ...

go tool trace trace.out

The trace viewer showed the P activity over time. What it revealed: two of the eight P’s were spending significant time in a tight loop — a goroutine doing JSON deserialization of large payloads, CPU-bound. Those two P’s were nearly 100% utilized. The other six P’s were cycling between idle and work-stealing, occasionally picking up goroutines from the overloaded P’s queues, but not consistently.

The overall CPU usage was high because two P’s were pegged at maximum. The latency was inconsistent because goroutines waiting in the global queue or in other P’s queues were being processed in bursts — whenever a work-steal happened to pull them into an active P.

The GOMAXPROCS was set to 8 (matching the EC2 instance’s 8 vCPUs). That was not the problem. The problem was that the deserializer was not parallelizable — each deserialization was a single goroutine doing sequential work on one P. Having 8 P’s did not help because the bottleneck was sequential work, not goroutine count.

The fix: decompose the deserialization work. Instead of one goroutine deserializing an entire payload, split the payload into chunks and process them concurrently. The work could then spread across multiple P’s.

// Before: one goroutine, one P
func deserializePayload(data []byte) ([]Record, error) {
    var records []Record
    return records, json.Unmarshal(data, &records)
}

// After: work spread across P's via concurrent chunk processing
func deserializePayload(ctx context.Context, data []byte) ([]Record, error) {
    // Split at record boundaries (JSON array elements)
    chunks := splitJSONArray(data, runtime.GOMAXPROCS(0))

    results := make([][]Record, len(chunks))
    errs := make([]error, len(chunks))
    var wg sync.WaitGroup

    for i, chunk := range chunks {
        wg.Add(1)
        go func(idx int, c []byte) {
            defer wg.Done()
            var recs []Record
            errs[idx] = json.Unmarshal(c, &recs)
            results[idx] = recs
        }(i, chunk)
    }

    wg.Wait()

    // collect results, check errors
    var all []Record
    for i, recs := range results {
        if errs[i] != nil {
            return nil, errs[i]
        }
        all = append(all, recs...)
    }
    return all, nil
}

This is a simplified version. The actual implementation handled JSON array splitting more carefully and used a semaphore to bound the chunk goroutines. The point is the structural change: from sequential work on one P to parallel work spread across multiple P’s.

After the change, the trace showed P utilization distributed more evenly. CPU was still high — the work was genuinely CPU-intensive — but latency became predictable because goroutines waiting for results were not competing with two overloaded P’s for scheduling time.

Reading Scheduler Behavior Without Reading Source Code

The source code reading was educational. For day-to-day debugging, the tooling is more practical.

**GODEBUG=schedtrace=1000**: prints scheduler state every 1000ms. Shows the number of goroutines in run queues, how many are idle, how many are running.

GODEBUG=schedtrace=1000 ./myservice
# Output every second:
# SCHED 1000ms: gomaxprocs=8 idleprocs=2 threads=23 spinningthreads=1 
#               idlethreads=14 runqueue=4 [3 0 1 0 2 0 0 1]

The numbers in brackets are the local run queue lengths per P. If one P consistently has a long queue while others are empty, work is not distributing evenly. That is the scheduler signal that something is wrong with how work is being created or structured.

**GODEBUG=scheddetail=1**: verbose per-goroutine scheduling info. Expensive. Useful for one-off debugging, not production.

Runtime trace: the most complete picture. Shows exactly what each P is doing over time, goroutine creation and destruction, blocking events, GC phases. The go tool trace UI makes it navigable. For latency investigations where pprof is not enough, the trace is the right tool.

Goroutine profile: go tool pprof http://localhost:6060/debug/pprof/goroutine shows all goroutines with their stacks. Useful for identifying goroutine leaks and unexpected blocking.

The Honest Limit of This Knowledge

Understanding the scheduler helps with a specific class of problems: CPU spikes that don’t match goroutine behavior, inconsistent latency under load, unexpected blocking patterns.

It does not help with everything. For most bugs — logic errors, off-by-one mistakes, incorrect business rule implementation — scheduler knowledge is irrelevant. The Go runtime’s scheduler is well-engineered and handles most workloads without requiring you to think about it.

The point of learning this is not to optimize preemptively. It is to have a framework when something does not make sense. The two-day debugging session I described at the start would have been shorter if I had understood that CPU contention could be a P-distribution problem rather than a goroutine count problem. That understanding did not require reading all of proc.go—it required understanding the M, P, G model and what work-stealing means for uneven workloads.

When the scheduler’s behavior contradicts your intuition, the contradiction is not random. It follows from the rules. Knowing the rules makes the contradiction interpretable.

Closing

Goroutines are lightweight threads is a useful approximation for getting started. It stops being useful when the system behaves in ways that the approximation doesn’t predict.

The scheduler is not magic. It is a specific algorithm: local queues per P, work-stealing from idle P’s, M’s blocked in syscalls handed off cleanly, preemption via signals. Each part of the algorithm has observable behavior if you know what to look for.

What I read in the source code is not exotic. It is the kind of runtime detail that, once you know it, makes debugging concurrency problems feel less like guesswork.

You do not debug concurrency by reading your code more carefully. You debug it by understanding how the runtime decided to execute it. The gap between those two things is where most scheduler-related confusion lives.