Building a Production-Grade Distributed Job Platform — BullMQ, Redis, Prometheus, and Grafana

From a polling-based PostgreSQL queue to a real distributed system with observability, retries, and CI/CD

In my previous article, I built a lightweight job processing system using PostgreSQL as the queue layer — intentionally avoiding external brokers to understand the fundamentals.

That system worked. But it had known limitations I left in deliberately:

• Polling-based worker — hits PostgreSQL every 2 seconds
• No distributed locking — two workers could pick the same job
• No queue broker — PostgreSQL was doing work it wasn’t designed for
• No observability — a console.log was the entire monitoring story

This article documents how I rebuilt that system properly — with BullMQ, Redis, Prometheus, Grafana, structured logging, rate limiting, JWT auth, and a full CI/CD pipeline.

Repository: https://github.com/imsaikatsen/distributed-job-platform

What changed — and why

The previous system used PostgreSQL as both a database and a queue. This works at small scale but breaks down quickly:

// previous approach — poll PostgreSQL every 2 seconds
setInterval(() => this.pollJobs(), 2000)

The problems this creates at scale are well understood. Polling frequency is a tradeoff between latency and database load. Two workers can read the same job before either updates its status. PostgreSQL was never designed to be a message broker.

BullMQ with Redis solves all three:

// new approach — BullMQ handles delivery, no polling needed
@Processor('job-processing')
export class JobProcessor extends WorkerHost {
  async process(job: Job): Promise<void> {
    // BullMQ calls this automatically when a job is available
  }
}

Redis is purpose-built for this. Jobs are stored in sorted sets, delivery is push-based not poll-based, and BullMQ handles distributed locking internally.

System architecture

The platform has two independent services that communicate exclusively through Redis:

API Service (Port 3000) → saves job to PostgreSQL → pushes job to BullMQ queue in Redis → returns job ID to client

Worker Service (Port 3001) → consumes jobs from Redis queue → processes them asynchronously → updates PostgreSQL with result

They never call each other directly. This is the key property of a distributed system — services are decoupled through a message broker.

Job lifecycle

Every job moves through a strict state machine:

PENDING → ACTIVE → COMPLETED
               ↘ FAILED → RETRYING → DEAD
PENDING → CANCELLED (manual)

The state is always the PostgreSQL row. Redis is the transport — not the source of truth. If Redis goes down, jobs are safe in PostgreSQL and can be re-queued on recovery.

This separation matters. It means the system can survive Redis restarts without data loss.

Reliability — exponential backoff and dead letter queue

This was the most important engineering decision. When a job fails, it should not retry immediately — that would hammer an already struggling downstream service.

BullMQ handles the timing automatically:

await this.jobQueue.add(jobType, jobData, {
  attempts: 3,
  backoff: {
    type:  'exponential',
    delay: 5000,
  },
})

The retry schedule becomes:

attempt 1 fails → wait 5s  → attempt 2
attempt 2 fails → wait 10s → attempt 3
attempt 3 fails → status: DEAD

This prevents the thundering herd problem — where hundreds of retrying clients make a recovering service worse. It is the same pattern AWS, Stripe, and every major distributed system uses.

Jobs that exhaust all retries move to a dead letter state. They stay in PostgreSQL permanently — queryable, auditable, and manually retryable by an admin.

Observability — the piece I was missing

The previous system had no observability. You could not answer basic questions like “how many jobs failed in the last hour?” or “what is the average processing time for payment jobs?”

I added three layers.

Structured logging with Pino

Every log is a JSON object with full context:

{
  "level": "info",
  "time": "2026-05-22T12:17:36.000Z",
  "service": "worker",
  "jobId": "f112e5be-85fc-4499-aa28-568c93eb7b2b",
  "type": "email_send",
  "duration": 2023,
  "attempt": 1,
  "msg": "Job completed successfully"
}

At 2am when something breaks, you can filter by jobId, type, or level instantly. Plain text logs do not give you this.

Prometheus metrics

The worker exposes these metrics at /metrics:

jobs_processed_total{type, status}   — total jobs by type and outcome
job_duration_ms{type, status}        — processing time histogram
worker_active_jobs{type}             — currently processing gauge
job_retries_total{type}              — retry counter

The API exposes:

jobs_created_total{type, priority}   — job creation counter

Grafana dashboard

Prometheus scrapes both services every 15 seconds. Grafana queries Prometheus and renders live graphs — queue depth, failure rate, processing time by job type

This answers the 2am question in seconds instead of minutes.

Authentication and rate limiting

The API uses JWT authentication with two tokens:

accessToken  → expires in 15 minutes — used for every API request
refreshToken → expires in 7 days     — used only to get a new access token

Short-lived access tokens limit the damage window if a token is stolen.

Rate limiting protects the API from abuse:

Login / Register → 10 requests per minute   (brute force protection)
Job creation     → 50 requests per minute   (queue flood prevention)
Global           → 100 requests per minute  (general abuse protection)

API documentation

The API is documented with Swagger UI — interactive, testable directly in the browser.

Every endpoint has request examples, response schemas, and authentication requirements clearly documented.

http://localhost:3000/api/v1/docs

Testing — 34 tests across two layers

I wrote two types of tests, each serving a different purpose.

Unit tests mock the database and test business logic in isolation:

it('should throw ForbiddenException if job belongs to another user', async () => {
  jobRepo.findOne.mockResolvedValue(mockJob)

  await expect(
    service.findOne('job-uuid-123', anotherUser),
  ).rejects.toThrow(ForbiddenException)
})

These run in under 2 seconds — no database needed. 24 unit tests cover UsersService, AuthService, and JobsService.

Integration tests spin up a real PostgreSQL instance and test the full HTTP flow:

it('should return 401 with invalid token', async () => {
  await request(app.getHttpServer())
    .get('/api/v1/auth/me')
    .set('Authorization', 'Bearer invalid-token')
    .expect(401)
})

10 integration tests cover the full authentication lifecycle.

CI/CD pipeline

Every push to main triggers three jobs in sequence:

Lint → Test (real PostgreSQL + Redis) → Build Docker Images

The test job spins up PostgreSQL and Redis as GitHub Actions services — the same way the app uses them in production. This catches integration failures before they reach deployment.

What this system can handle

A single worker instance processes jobs sequentially. But because the API and Worker are fully decoupled through Redis, horizontal scaling is straightforward — run multiple worker containers against the same queue and BullMQ distributes jobs automatically with no race conditions.

The Prometheus metrics make scaling decisions data-driven. When worker_active_jobs consistently hits the concurrency limit and job_duration_ms increases, that is the signal to add another worker.

What I would add next

• Refresh token rotation with database storage and revocation
• Job scheduling — recurring jobs with cron expressions
• Admin endpoints — retry dead jobs, view all users
• Deployment to a cloud provider with the full Docker Compose stack

Key lessons

PostgreSQL as a queue works — until it doesn’t. It is a valid starting point for learning but the operational costs grow quickly. BullMQ with Redis is purpose-built and handles the hard distributed systems problems for you.

Observability is not polish. Structured logging and metrics should be designed in from the start, not added later. The 2am debugging experience is completely different with and without them.

Testing at two layers matters. Unit tests catch business logic bugs fast. Integration tests catch the bugs that only appear when real services interact. Both are necessary.

🔗 Repository: https://github.com/imsaikatsen/distributed-job-platform

🔗 Previous article: https://medium.com/@sensaikatcse/designing-a-lightweight-asynchronous-job-processing-system-from-first-principles-377c50ceb5dc