I Ignored Backpressure in My Go Worker and the Queue Became a Memory Problem

The worker looked reliable while traffic was calm, but once jobs arrived faster than they could be processed, the queue quietly started consuming the service.

AI once helped me produce Go code that looked complete until production behavior exposed what was missing. I wrote about that lesson in AI Wrote My Go Code, But Production Taught Me the Real Lesson. You can read it here:

[embed]AI Wrote My Go Code, But Production Taught Me the Real Lesson The code compiled, the feature worked in staging, but production exposed every shortcut I ignored.renaldid.medium.com

The worker in this story had the same kind of problem.

Its code was readable, tests were green, and every submitted job eventually reached a processor during local testing. What I had not tested was the relationship between arrival speed and processing speed.

Production sent jobs faster than the worker could finish them.

Instead of slowing producers down or rejecting excess work, my service kept accepting everything. The queue grew, payloads remained reachable in memory, and latency increased long before the worker reported a clear failure.

I had built a queue without backpressure.

The Queue That Could Grow Forever

The first implementation stored incoming jobs in a slice protected by a mutex.

type UnboundedQueue struct {
    mu sync.Mutex
    jobs[] SettlementJob
}

func(q * UnboundedQueue) Enqueue(job SettlementJob) {
    q.mu.Lock()
    q.jobs = append(q.jobs, job)
    q.mu.Unlock()
}

Nothing stopped the slice from growing.

If the worker processed 50 jobs per second while the API submitted 200, the difference remained in memory. Each job also carried a payload, so queue depth was not the only concern. The retained bytes mattered too.

The garbage collector could not release those payloads because the queue still referenced them. This was not a GC failure. It was an admission-control failure.

The service needed to decide how much pending work it was willing to own.

Giving the Queue a Real Capacity

I replaced the growing slice with a bounded channel and made queue saturation visible to callers.

package main

import (
    "context"
    "encoding/json"
    "errors"
    "log"
    "net/http"
    "os"
    "os/signal"
    "runtime"
    "strconv"
    "strings"
    "sync"
    "sync/atomic"
    "syscall"
    "time"
)

var ErrQueueFull = errors.New("settlement queue is full")

type SettlementJob struct {
    ID string
    Payload[] byte
}

type SettlementQueue struct {
    jobs chan SettlementJob
    wg sync.WaitGroup
    accepted atomic.Int64
    rejected atomic.Int64
    processed atomic.Int64
    bufferedByte atomic.Int64
}

func NewSettlementQueue(ctx context.Context, workers, capacity int) * SettlementQueue {
    queue: = & SettlementQueue {
        jobs: make(chan SettlementJob, capacity),
    }
    for workerID: = 1;workerID <= workers;workerID++{
        queue.wg.Add(1)
        go queue.worker(ctx, workerID)
    }
    return queue
}

The submission method does not wait forever for queue capacity. It accepts the job immediately when space exists, respects request cancellation, or returns a clear overload error.

func(q * SettlementQueue) Submit(ctx context.Context, job SettlementJob) error {
    select {
        case <-ctx.Done():
            return ctx.Err()
        case q.jobs < -job:
            q.accepted.Add(1)
            q.bufferedByte.Add(int64(len(job.Payload)))
            return nil
        default:
            q.rejected.Add(1)
            return ErrQueueFull
    }
}

Returning an error felt uncomfortable at first. I wanted every job to be accepted.

Production taught me that accepting work without enough capacity is not reliability. It only delays the failure while consuming more memory.

The Workers Had Their Own Time Budget

A bounded queue controls how much work may wait. The workers still need limits on how long one job can occupy capacity.

func(q * SettlementQueue) worker(ctx context.Context, workerID int) {
    defer q.wg.Done()
    for {
        select {
            case <-ctx.Done():
                return
            case job:
                = < -q.jobs:
                    jobCtx, cancel: = context.WithTimeout(ctx, 500 * time.Millisecond)
                err: = processSettlement(jobCtx, job)
                cancel()
                q.bufferedByte.Add(-int64(len(job.Payload)))
                if err != nil {
                    log.Printf("worker=%d job=%s error=%v", workerID, job.ID, err)
                    continue
                }
                q.processed.Add(1)
        }
    }
}

func processSettlement(ctx context.Context, job SettlementJob) error {
    select {
        case <-time.After(120 * time.Millisecond):
            return nil
        case <-ctx.Done():
            return ctx.Err()
    }
}

In the real service, processSettlement called a database and an external provider. Both operations received the job context so one slow dependency could not hold a worker forever.

Queue Depth Became an Operational Signal

I also exposed the queue state because a bounded queue is only useful when saturation can be observed.

type QueueStats struct {
    Depth int `json:"depth"`
    Capacity int `json:"capacity"`
    Accepted int64 `json:"accepted"`
    Rejected int64 `json:"rejected"`
    Processed int64 `json:"processed"`
    BufferedBytes int64 `json:"buffered_bytes"`
    HeapAllocatedMB uint64 `json:"heap_allocated_mb"`
}

func(q * SettlementQueue) Stats() QueueStats {
    var mem runtime.MemStats
    runtime.ReadMemStats( & mem)
    return QueueStats {
        Depth: len(q.jobs),
        Capacity: cap(q.jobs),
        Accepted: q.accepted.Load(),
        Rejected: q.rejected.Load(),
        Processed: q.processed.Load(),
        BufferedBytes: q.bufferedByte.Load(),
        HeapAllocatedMB: mem.HeapAlloc / 1024 / 1024,
    }
}

Queue depth, rejected jobs, and buffered bytes became more useful than a generic “worker is running” health check. They showed whether processing capacity was keeping up with incoming work.

Applying Backpressure at the HTTP Boundary

The API translated queue saturation into a controlled response instead of hiding it.

func submitHandler(queue * SettlementQueue) http.HandlerFunc {
    return func(w http.ResponseWriter, r * http.Request) {
        payloadKB: = 64
        if raw: = r.URL.Query().Get("payload_kb");raw != "" {
            if value, err: = strconv.Atoi(raw);
            err == nil && value > 0 && value <= 256 {
                payloadKB = value
            }
        }
        job: = SettlementJob {
            ID: strconv.FormatInt(time.Now().UnixNano(), 36),
            Payload: [] byte(strings.Repeat("x", payloadKB * 1024)),
        }
        if err: = queue.Submit(r.Context(), job);err != nil {
            writeJSON(w, http.StatusServiceUnavailable, map[string] string {
                "error": err.Error(),
            })
            return
        }
        writeJSON(w, http.StatusAccepted, map[string] string {
            "job_id": job.ID,
        })
    }
}

The service setup kept the queue intentionally small so saturation could be reproduced easily.

func main() {
    ctx, stop: = signal.NotifyContext(
        context.Background(),
        os.Interrupt,
        syscall.SIGTERM,
    )
    defer stop()

    queue: = NewSettlementQueue(ctx, 4, 32)
    mux: = http.NewServeMux()
    mux.HandleFunc("POST /settlement-jobs", submitHandler(queue))
    mux.HandleFunc("GET /stats", func(w http.ResponseWriter, r * http.Request) {
        writeJSON(w, http.StatusOK, queue.Stats())
    })
    server: = & http.Server {
        Addr: "127.0.0.1:8080",
        Handler: mux,
        ReadHeaderTimeout: 2 * time.Second,
    }
    go func() {
        log.Println("server running on http://127.0.0.1:8080")
        if err: = server.ListenAndServe();
        err != nil &&
            !errors.Is(err, http.ErrServerClosed) {
                log.Printf("server error: %v", err)
                stop()
            }
    }() < -ctx.Done()
    shutdownCtx, cancel: = context.WithTimeout(context.Background(), 3 * time.Second)
    defer cancel()

    _ = server.Shutdown(shutdownCtx)
    queue.wg.Wait()
}

func writeJSON(w http.ResponseWriter, status int, body any) {
    w.Header().Set("Content-Type", "application/json")
    w.WriteHeader(status)
    _ = json.NewEncoder(w).Encode(body)
}

What Changed in Production

The queue stopped behaving like unlimited storage.

When workers had capacity, jobs were accepted normally. When the queue filled, producers received a clear overload response and could retry later with backoff. For critical jobs that could not be rejected, we moved the workload to a durable message broker instead of pretending an in-memory queue was reliable storage.

Memory became more predictable because pending work had a fixed upper bound. Queue saturation also became visible before the entire service slowed down.

Backpressure did not increase processing capacity. It prevented demand from silently exceeding that capacity.

Final Thought

The worker was never the only problem.

The service kept accepting jobs even after the worker had fallen behind. Every accepted payload became another promise stored in memory, with no limit on how many promises the process could hold.

A queue should not only move work between goroutines. It should express capacity.

Once I added that boundary, overload stopped looking like mysterious memory growth. It became a clear operational state the service could measure, report, and handle.

Sometimes reliability means accepting more work.

Sometimes it means having the discipline to say the queue is full.