Fixed Worker Threads are bottleneck sometimes ? Go and Java

Why “Fixed Workers” are often a bottleneck, and how to use Semaphores to build high-throughput systems like LeetCode

Imagine you’re building a code execution engine — something like LeetCode. You have a stream of user submissions coming in via RabbitMQ, and you need to spin up Docker containers to run them.

Most developers start with a Fixed Worker Pool. They spin up 10 workers, each pulling one message at a time. It feels safe. It feels organised. But it’s likely killing your performance.

The “I/O Bound” Trap

In an orchestrator (like a Go or Java app managing Docker), your threads spend 99% of their time waiting. They wait for the Docker API, they wait for the container to finish, they wait for the network.

If you have 10 workers and a 16-core server, your CPU is likely sitting idle while your users wait in a long queue. In the world of I/O, 10 workers is a bottleneck, not a limit.

The Pattern: The Dispatcher & The Semaphore

To solve this, we move from “Pulling” to “Dispatching.” Instead of a worker sitting on a message until it’s done, we treat workers as Dispatchers.

1. The Dispatcher: Receives a message and immediately spawns a background unit of work (a Goroutine in Go or a Virtual Thread in Java).
2. The Semaphore: Since we can’t spawn infinite Docker containers without crashing our server, we use a Semaphore!

Whether you use a chan struct{} in Go or a java.util.concurrent.Semaphore in Java, the logic is the same: Decouple your threads from your tasks.

Why semaphore?

— Backpressure

It acts as a signal to the rest of your system.

When the Semaphore is full, the virtual threads/goroutines block. This “pressure” naturally slows down the rate at which you pull from RabbitMQ. This is called Implicit Backpressure, and it prevents your application from trying to do more than it is physically capable of.

Example piece of code in Go

// this function is responsible for initiating a new worker thread which 
// eventually works as a dispatcher
func (pool *SubmissionWorkerPool) startWorker(workerID int) {
    defer pool.wg.Done()

    // taskWg tracks background goroutines to ensure graceful shutdown
    var taskWg sync.WaitGroup 

    for {
        select {
        case <-pool.ctx.Done():
            log.Printf("Worker %d: Shutting down. Waiting for active tasks...", workerID)
            taskWg.Wait() // Don't exit until all containers finish
            return

        case message, ok := <-messageChan:
            if !ok { return }

            // DISPATCH: Hand off work to a background goroutine immediately
            taskWg.Add(1)
            go func(msg amqp.Delivery) {
                defer taskWg.Done()

                // THROTTLE: Acquire a permit from the semaphore
                // If the buffer is full, this line blocks (Implicit Backpressure)
                pool.semaphore <- struct{}{}

                // RELEASE: Ensure the spot is freed when the container exits
                defer func() { <-pool.semaphore }()

                // EXECUTE: Call the Docker service
                if err := pool.processRawSubmission(workerID, msg.Body); err != nil {
                    msg.Nack(false, false) // Reject on failure
                } else {
                    msg.Ack(false) // Success!
                }
            }(message)
        }
    }
}

The Final Verdict

Whether writing in Go or Java, the goal of a better “Engineer” is to maximize resource utilization while preventing system collapse. By separating the receipt of a message from the execution of the task, and guarding the execution with a Semaphore, we can build systems that are both incredibly fast and resource optimised.