Building a Scalable Distributed Docker build system from Scratch

Before we begin: This is the repository Link: https://github.com/vky5/Blacktree

Objective

Building a distributed orchestrator-worker system for building Docker images. Workers can scale horizontally, and failure of one doesn’t halt the entire system.

Hello knight. Today you are going on one of the most difficult quests: understanding one of the most basic yet complex examples of a distributed system. But what if a knight doesn’t have any other objective? Your task is to take a message with you — a message that looks something like this:

{
  "jobId": "unique-id-123",
  "repoUrl": "user/reponame",
  "branch": "main",
  "dockerfilePath": "Dockerfile",
  "contextDir": ".",
  "imageName": "myapp:latest",
  "githubToken": "<token>"
}

Don’t worry if you don’t know what this even is. You’re going to find out a lot of things: what a Job is, and what this message is for.

Message sent from backend (Knight goes from Palace)

Like a palace for the king, the main backend takes every decision. Whether to send a knight with a message in the first place, what is written in the message, managing the data inside the message — everything.

A message in JSON format travels from the backend. It contains all the instructions a worker will need to build a Docker image: which repo to clone, which branch, which Dockerfile to use, and even the secret token for private repos.

Let me give you a brief overview of what each field is:

• jobId — to track the message, like a unique mail ID
• repoUrl — the repository link (user/reponame)
• branch — which branch to clone
• dockerfilePath — path to the Dockerfile
• contextDir — directory in which the Dockerfile resides
• imageName — the final Docker image name
• githubToken — for private repo access

A narrow road (RabbitMQ)

As soon as the knight receives the message, he marches down a road. The road is very narrow; only one horse can go at a time. There he meets many other knights sent by the king with similar messages, all standing one after another.

This road is the RabbitMQ Messaging Queue — a single-lane, first-in-first-out system.

There are actually two parallel roads (queues) here:

1. backend → orchestrator — messages carrying jobs
2. orchestrator → backend — job status updates

Each message carries a “routing key,” a magical permission scroll telling the message which road it can take. This is called a Direct Exchange.

Example: Backend pushing message in RabbitMQ (NestJS snippet)

// method to publish message to the queue
  publishMessage(routingKey: string, message: PublishDeploymentMessageDto) {
    if (!this.channel) {
      throw new Error('RabbitMQ channel is not initialized');
    }
    try {
      const buffer = Buffer.from(JSON.stringify(message));
      const published = this.channel.publish(
        this.exchange, // exchange name
        routingKey, // routing key
        buffer, // message buffer
        {
          persistent: true, // ensures message is saved to disk
        },
      );

      if (!published) {
        throw new Error('Message could not be published');
      }
      return {
        message: `Message published to exchange "${this.exchange}" with routing key "${routingKey}"`,
      };
    } catch (error) {
      console.log('Error publishing message:', error);
      throw new Error(
        `Failed to publish message: ${error instanceof Error ? error.message : 'Unknown error'}`,
      );
    }
  }

Preparing and sending the messaging

// Prepare and send build message to orchestrator queue
const message: PublishDeploymentMessageDto = {
    deploymentId: existingVersionId,
    token: deployment.user.token,
    repository: deployment.repository,
    branch: deployment.branch,
    dockerFilePath: deployment.dockerFilePath,
    contextDir: deployment.contextDir,
    createdAt: new Date().toISOString(),
  };
this.messageingQueueService.publishMessage('blacktree.routingKey', message);

Here, the backend (king) sends the knight with a message to the orchestrator queue. The knight cannot deviate — the direct exchange ensures he follows the right path.

What the knight understands

There is a giant gatekeeper (exchange) that checks if the knight has permission to travel on the single road called execute.queue. Only messages with the right routing key can pass.

• Exchange: blacktree.direct
• Routing Key: orchestrator.execute
• Queue: execute.queue
• Producer: backend (API)
• Consumer: orchestrator

The knights now wait in line, ready to be picked up by the orchestrator when it’s their turn to build somethi

Enter the Worker Knight

In a nearby village, another set of knights is preparing for their mission. These are the workers. Unlike the message-carrying knights, workers are the ones who do the heavy lifting: they build the Docker images, push them to the registry, and clean up after themselves.

Worker roles are:

• Cloning the repository
• Building the Docker image
• Uploading the image to AWS ECR (Elastic Container Registry)
• Cleaning up the cloned repository
• Sending status back to the orchestrator castle

Workers are not idle; they live in their own castle but always look for jobs. Each worker wants to register with the orchestrator so they can be picked up when a knight arrives with a message.

Worker Registration — Joining the Orchestrator Castle

Before a worker can start its quest, it must knock on the orchestrator’s gate and introduce itself. This is done via gRPC, a magical scroll that allows instant, structured communication.

// worker main.go snippet
go grpc.StartGRPCServer(port, workerID) // worker starts its gRPC server ready to execute any command
time.Sleep(500 * time.Millisecond)
success := registerWithOrchestrator(workerID, orchestratorAddr, ip, port, "us-east-1")
if !success {
    log.Fatal("❌ Registration with orchestrator failed, exiting...")
}

Here’s what happens step by step:

1. Worker starts its own gRPC server — like opening the gates to its village so the orchestrator can communicate.
2. Worker registers itself with the orchestrator by sending its ID, IP, port, and region.
3. Orchestrator acknowledges the registration, adding the worker to its pool of available knights ready for a mission.

The Orchestrator’s perspective

When a worker arrives, the orchestrator does the following:

• Receives the registration request
• Stores worker information in its worker manager (a guarded scrollbook)
• Marks the worker as FREE, placing it in the pool of available knights ready for job assignments

func (s *OrchestratorGRPCServer) Register(ctx context.Context, req *workerpb.WorkerInfo) (*workerpb.RegisterAck, error) {
    log.Printf("Received Register request from worker ID: %s", req.Id)
    err := s.Manager.RegisterWorker(ctx, req) // adds worker to pool
    if err != nil {
        return nil, err
    }
    return &workerpb.RegisterAck{Success: true}, nil
}

Thanks to Go’s concurrency magic, every registration request is handled in a separate goroutine. This means multiple workers can register at the same time without blocking each other.

• Each worker is independently added to the free worker channel.
• The orchestrator always knows which knights are ready for the next quest.

Storytelling perspective

The worker knight has now arrived at the orchestrator castle and pledged loyalty. The knight is ready to be assigned any message-carrying task the orchestrator sends. The line of messages (from RabbitMQ) is getting ready, and soon, the orchestrator will pair each message with a free worker knight to complete the mission.

Worker Job Life Cycle — From Receiving a Message to Returning the Result

Our worker knight is now fully registered with the orchestrator castle. The next step is the actual quest: receiving a message, completing the task, and reporting back.

Step 1: Receiving a Job

The orchestrator pairs a free worker with a queued message. This is done via gRPC call RunJob. The worker checks if it’s free and then starts the quest.

func (w *WorkerGRPCServer) RunJob(ctx context.Context, req *jobpb.JobRequest) (*jobpb.JobResponse, error) {
    w.mu.Lock()
    if w.isBusy {
        w.mu.Unlock()
        return nil, fmt.Errorf("worker is currently busy")
    }
    w.isBusy = true
    w.mu.Unlock()
defer func() {
        w.mu.Lock()
        w.isBusy = false
        w.mu.Unlock()
    }()
    return RunJobLogic(ctx, req)
}

• The worker locks itself to ensure no other job is assigned simultaneously.
• RunJobLogic handles the full sequence of building, pushing, and cleaning.

Step 2: Cloning the Repository

folder, err := repo.CloneRepo(repo.CloneRepoInput{
    RepoURL: req.RepoUrl,
    Branch:  req.Branch,
    Token:   &req.GithubToken,
})

The worker fetches the source code like a knight picking up tools for the quest.

Step 3: Building the Docker Image

err = builder.BuildImage(builder.BuildImageOptions{
    ImageName:      req.ImageName,
    ContextDir:     req.ContextDir,
    DockerfilePath: req.DockerfilePath,
}, *folder)

The worker forges the Docker image from the cloned repo.

Step 4: Uploading to AWS ECR

err = aws.LoginDockerToAWS()
err = builder.TagAndPushImage(ctx, aws.DockerCli, req.ImageName, *aws.Credentials)

After building, the worker knight carries the treasure (Docker image) to the AWS registry castle (ECR) and stores it securely.

Step 5: Returning Response

After the quest, the worker reports back to the orchestrator castle with the results, success or failure, and any logs.

return &jobpb.JobResponse{
    JobId:    req.JobId,
    Success:  true,
    Logs:     "Build and push successful",
    ImageUrl: ecrURL,
    Error:    "",
}, nil

Finally, the worker knight becomes FREE again and rejoins the pool, ready for the next message.

Storytelling perspective

The knight has completed the quest, returned to the orchestrator castle, and reported the treasure’s location (Docker image URL). The orchestrator now forwards this information to the backend kingdom, so the king knows the mission succeeded.

Multiple workers perform these quests in parallel, each independently handling messages without stepping on each other’s path.

Worker Life Cycle Diagram

Orchestrator — The Master of the Castle

The orchestrator is like the castle commander, coordinating all knights (workers) and ensuring that messages from the kingdom (backend) are delivered and executed correctly.

Orchestrator Responsibilities

1. Receive Messages from RabbitMQ

• The orchestrator listens on a queue (execute.queue) using Direct Exchange.
• Each message has a routing key determining the queue it can traverse.
• Multiple messages can arrive concurrently, so the orchestrator processes them in a pull-based manner.

2. Register and Manage Workers

• Workers register via gRPC (Register()) when they come online.
• Each worker is added to the free workers channel (chan *Worker) so jobs can be assigned.
• Worker metadata (IP, port, ID, region) is stored in a mutex-protected map.

3. Assign Jobs to Free Workers

• The orchestrator maintains a queue of pending jobs.
• Whenever a worker becomes free, it pops a job from the queue and assigns it via gRPC.
• Concurrency is handled using Go’s goroutines, channels, and mutexes.

4. Health Checks

• A background goroutine periodically pings each worker via Ping().
• If a worker fails a health check, it is removed from the free channel, and any in-progress job is requeued.

5. Report Results Back to Backend

• Once a job completes (success or failure), the orchestrator sends the response back through RabbitMQ (status.queue).
• Backend can trigger a manual retry if needed.

Detailed Code Flow

1. Worker Registration

func (s *OrchestratorGRPCServer) Register(ctx context.Context, req *workerpb.WorkerInfo) (*workerpb.RegisterAck, error) {
    log.Printf("Register request from worker ID: %s", req.Id)

    s.Manager.RegisterWorker(ctx, req) // updates internal map and pushes worker to free channel

    return &workerpb.RegisterAck{Success: true}, nil
}

Notes:

• RegisterWorker() updates the workers map mutex-protected to prevent concurrent writes.
• Worker is immediately added to freeWorkerChan, making it ready for assignments.

2. Job Dispatching Logic

func (d *JobDispatcher) Start() {
    for job := range d.jobQueue {
        go func(job Job) {
            worker := <-d.freeWorkerChan  // blocks until a worker is available
            ctx, cancel := context.WithTimeout(context.Background(), 10*time.Minute)
            defer cancel()
res, err := worker.Client.RunJob(ctx, job.Request)
            if err != nil {
                log.Printf("Job %s failed: %v", job.Request.JobId, err)
                d.jobQueue <- job  // requeue failed job
            } else {
                d.sendResultToBackend(res)
            }
            d.freeWorkerChan <- worker  // mark worker free again
        }(job)
    }
}

Concurrency Handling:

• Each job is assigned in its own goroutine, allowing multiple workers to execute in parallel.
• freeWorkerChan ensures only free workers are assigned jobs.
• If a worker fails, the job is requeued automatically.

3. Health Checks

func (m *WorkerManager) StartHealthChecks() {
    ticker := time.NewTicker(30 * time.Second)
    for range ticker.C {
        for _, w := range m.GetAllWorkers() {
            go func(worker *Worker) {
                ctx, cancel := context.WithTimeout(context.Background(), 5*time.Second)
                defer cancel()
                res, err := worker.Client.Ping(ctx, &jobpb.PingRequest{})
                if err != nil || res.Status != jobpb.WorkerStatus_FREE {
                    log.Printf("Worker %s failed health check", worker.ID)
                    m.RemoveWorker(worker.ID)
                }
            }(w)
        }
    }
}

Key Points:

• Non-blocking: Each ping runs in its own goroutine.
• Workers failing ping are removed and jobs reassigned.
• Keeps system resilient to failures.

4. RabbitMQ Integration

func (d *JobDispatcher) ListenRabbitMQ() {
    msgs := rabbitmq.Consume("execute.queue")
    for m := range msgs {
        job := parseJob(m.Body)
        d.jobQueue <- job
    }
}

• Messages are consumed from execute.queue in real-time.
• Dispatcher pushes jobs into jobQueue channel for workers.

Lifecycle Diagram

Explanation:

• The diagram shows full flow from backend to worker and back.
• Concurrency is handled by goroutines and channels (jobQueue, freeWorkerChan).
• Health checks run in parallel to ensure stability.

Integration — How Messages, Orchestrator, and Workers Work Together

Now that the messages are queued on the RabbitMQ road, and our worker knights are registered at the orchestrator castle, it’s time to see how all of them interact in one grand quest. This is where the backend kingdom, the orchestrator castle, and the worker knights come together.

Step 1: Messages Arrive

• The backend sends a job message to RabbitMQ (execute.queue).
• Each message has a routing key, ensuring it travels only to the orchestrator.
• Messages wait patiently in the queue until a worker is free.

Step 2: Dispatcher Assigns Jobs

• The orchestrator’s dispatcher goroutine consumes messages from the queue.
• It looks into the pool of free worker knights maintained by the worker manager.
• If a free knight is available, the dispatcher assigns the job via gRPC.

Step 3: Worker Executes the Job

• The worker knight locks itself to prevent multiple jobs simultaneously.
• It clones the repository, builds the Docker image, uploads it to AWS ECR, cleans up, and prepares the result.

Step 4: Result Handling

• The result handler receives the job outcome from the worker.
• It pushes the response back to the backend through RabbitMQ (status.queue).
• The worker becomes free again, ready for the next message.

Step 5: Health Checks Run in Parallel

• Meanwhile, the health checker goroutine continuously pings all workers.
• Any failing worker is temporarily removed, and their in-progress job is requeued.

Connecting All Parts

In this diagram, you can see how the orchestrator coordinates the knights. The dispatcher consumes jobs from RabbitMQ, the worker manager assigns them to free knights via gRPC, the health checker ensures readiness, and the result handler delivers updates back to the kingdom.