When Cron Misses, Revenue Drops — Building a Resilient Job Queue with Dedupe, Backoff, and Dead-Letter Triage

Ensuring revenue stability through intelligent job management and robust error handling — with working PHP code for every pattern.

Photo by NHN on Unsplash

Imagine your SaaS platform charges subscription renewals every night at midnight. The cron job fires, calls the payment API, processes 2,400 customers, and everything looks fine. Except the payment API was intermittently returning 503s between 12:04 and 12:11. Thirty-seven customers didn’t get charged. Your cron job logged no errors — the failures were swallowed silently. The cron job marked itself complete.

Nobody knows. The customers don’t know. Finance doesn’t know. The monitoring dashboard shows green.

You find out three weeks later when a customer asks why they haven’t been billed.

This is the specific failure mode that cron jobs produce: silent, delayed, revenue-impacting. The problem is not that cron jobs fail — every system has failures. The problem is that vanilla cron has no built-in mechanism to retry failures, prevent duplicate execution, or flag jobs that need human attention. When it fails, it fails silently, and silence is the most expensive kind of failure.

This article covers the three patterns that prevent this: deduplication, exponential backoff with jitter, and dead-letter queues. Each comes with complete, production-ready PHP code — not pseudocode or concept sketches, but implementations you can adapt and deploy.

The Fundamental Problem With Cron

Cron’s execution model is simple: fire the command at the scheduled time. That’s all it does.

It does not:

• Know whether the previous run succeeded or failed
• Retry failed jobs automatically
• Prevent duplicate runs if the previous job is still running
• Track which jobs need human attention
• Give you any visibility into what happened

These are not edge cases — they are the operational realities of any system running under real load. External APIs go down. Databases hit connection limits. Network partitions happen. The cron job that “always works” has never been tested under these conditions at 2am on a Friday.

The solution is not to replace cron (cron is fine for triggering jobs) but to wrap it with a job queue system that provides the guarantees cron itself lacks: exactly-once execution, intelligent retry, and graceful failure handling.

The Schema: Foundation for All Three Patterns

All three patterns share a common database schema. Design this correctly first and the rest follows:

CREATE TABLE jobs (
    id              BIGINT UNSIGNED AUTO_INCREMENT PRIMARY KEY,
    queue           VARCHAR(50)    NOT NULL DEFAULT 'default',
    payload         JSON           NOT NULL,
    status          ENUM(
                        'pending',
                        'processing',
                        'completed',
                        'failed',
                        'dead'
                    ) NOT NULL DEFAULT 'pending',
    idempotency_key VARCHAR(255)   NULL,      -- for deduplication
    attempts        TINYINT UNSIGNED NOT NULL DEFAULT 0,
    max_attempts    TINYINT UNSIGNED NOT NULL DEFAULT 5,
    available_at    DATETIME       NOT NULL DEFAULT CURRENT_TIMESTAMP,
    reserved_at     DATETIME       NULL,      -- set when a worker picks up the job
    completed_at    DATETIME       NULL,
    failed_at       DATETIME       NULL,
    last_error      TEXT           NULL,      -- last exception message
    created_at      DATETIME       NOT NULL DEFAULT CURRENT_TIMESTAMP,
    updated_at      DATETIME       NOT NULL DEFAULT CURRENT_TIMESTAMP ON UPDATE CURRENT_TIMESTAMP,
    UNIQUE INDEX idx_idempotency (idempotency_key),
    INDEX idx_queue_status_available (queue, status, available_at),
    INDEX idx_status_reserved (status, reserved_at)
);
CREATE TABLE dead_letter_jobs (
    id              BIGINT UNSIGNED AUTO_INCREMENT PRIMARY KEY,
    original_job_id BIGINT UNSIGNED NOT NULL,
    queue           VARCHAR(50)    NOT NULL,
    payload         JSON           NOT NULL,
    failure_reason  TEXT           NOT NULL,
    attempts        TINYINT UNSIGNED NOT NULL,
    moved_at        DATETIME       NOT NULL DEFAULT CURRENT_TIMESTAMP,
    reviewed_at     DATETIME       NULL,      -- set when a human has looked at it
    resolution      TEXT           NULL,      -- what was done about it
    created_at      DATETIME       NOT NULL DEFAULT CURRENT_TIMESTAMP
);

Key design decisions:

The idempotency_key column with a UNIQUE index is what makes deduplication atomic — a UNIQUE constraint violation on insert is the database enforcing exactly-once semantics, not application code trying to check first.

The available_at column is the backoff mechanism — instead of sleeping between retries, update available_at to a future time. The worker query filters on available_at <= NOW(), so the job naturally becomes available again when the backoff window expires.

The reserved_at column enables detecting stuck jobs — a job that has been in processing status for longer than the expected execution time has likely crashed without updating its status.

Pattern 1: Deduplication — Exactly-Once Execution

Imagine a payment capture job triggered both by a webhook (payment provider notifying you) and a scheduled retry (your system retrying after a suspected failure). Both triggers arrive within seconds of each other. Without deduplication, you capture the payment twice. One customer is charged twice. One chargeback arrives.

Deduplication prevents this by assigning a stable, unique key to each logical job — a key that is the same regardless of how many times the job is submitted. The database enforces that only one job with that key can exist in a non-terminal state.

<?php
final class JobQueue
{
    public function __construct(private readonly \PDO $pdo) {}
    /**
     * Enqueue a job with idempotency - safe to call multiple times for the same logical job.
     *
     * @param string $queue       Which queue this job belongs to
     * @param array  $payload     The job data
     * @param string $idempotencyKey Stable unique key for this logical operation
     *                               e.g. "payment-capture:{$orderId}" or md5(serialize($payload))
     * @param int    $maxAttempts Maximum retry attempts before moving to dead letter queue
     *
     * @return int|null The job ID if created, null if the job already exists (idempotent)
     */
    public function enqueue(
        string $queue,
        array  $payload,
        string $idempotencyKey,
        int    $maxAttempts = 5
    ): ?int {
        try {
            $this->pdo->prepare("
                INSERT INTO jobs (queue, payload, idempotency_key, max_attempts)
                VALUES (:queue, :payload, :idempotency_key, :max_attempts)
            ")->execute([
                'queue'           => $queue,
                'payload'         => json_encode($payload, JSON_THROW_ON_ERROR),
                'idempotency_key' => $idempotencyKey,
                'max_attempts'    => $maxAttempts,
            ]);
            return (int) $this->pdo->lastInsertId();
        } catch (\PDOException $e) {
            // Unique constraint violation on idempotency_key = job already exists
            // This is the correct behavior - not an error
            if ($e->getCode() === '23000') {
                return null; // Signal: already queued, nothing to do
            }
            throw $e;
        }
    }
    /**
     * Generate a stable idempotency key from job parameters.
     * The key must be identical for the same logical operation, regardless of when it's called.
     */
    public static function makeIdempotencyKey(string $jobType, mixed ...$identifiers): string
    {
        return $jobType . ':' . md5(implode(':', array_map('strval', $identifiers)));
    }
}

Usage — the same job can be submitted as many times as needed, with no duplicate created:

$queue = new JobQueue($pdo);
// Called by webhook handler
$queue->enqueue(
    queue:           'payments',
    payload:         ['order_id' => 4821, 'amount' => 9999, 'currency' => 'USD'],
    idempotencyKey:  JobQueue::makeIdempotencyKey('payment-capture', 4821),
    maxAttempts:     3
);
// Called by scheduled retry 5 seconds later - does nothing, returns null
$queue->enqueue(
    queue:           'payments',
    payload:         ['order_id' => 4821, 'amount' => 9999, 'currency' => 'USD'],
    idempotencyKey:  JobQueue::makeIdempotencyKey('payment-capture', 4821),
    maxAttempts:     3
);

The UNIQUE constraint on idempotency_key makes this atomic. There is no window between a "check if exists" SELECT and an INSERT where a race condition can create two jobs — the database constraint prevents it at the hardware level.

Choosing idempotency keys correctly is the critical design decision:

• "payment-capture:{$orderId}" — safe, one capture attempt per order
• "subscription-renewal:{$userId}:{$month}" — one renewal per user per billing period
• "email-confirmation:{$userId}:{$emailType}" — one email of a given type per user
• md5(serialize($payload)) — last resort when no natural identifier exists; fragile if payload changes

A key that is too broad ("daily-report") prevents running the same job type twice in a day even if you need to. A key that is too narrow (includes a timestamp) defeats the purpose — every submission creates a new unique key.

Pattern 2: Exponential Backoff With Jitter — Intelligent Retry

A job fails. Should you retry immediately? Almost never. The most common failure causes — external API rate limits, temporary service unavailability, database connection exhaustion — get worse under immediate retry pressure. Hammering a struggling service with immediate retries can turn a 30-second outage into a 10-minute one.

Exponential backoff increases the wait between retries geometrically. Jitter adds randomness to prevent synchronized retries — if 50 jobs all hit their first retry at exactly the same moment, they create a synchronized load spike (the “thundering herd”). Randomizing within a range spreads that load.

<?php
final class JobWorker
{
    // Base delay in seconds - first retry waits this long
    private const BASE_DELAY_SECONDS = 60;
    // Maximum delay cap - retries never wait longer than this
    private const MAX_DELAY_SECONDS  = 3600; // 1 hour
    // Jitter factor: delay varies ±25% of calculated value
    private const JITTER_FACTOR      = 0.25;
    public function __construct(
        private readonly \PDO    $pdo,
        private readonly string  $queue = 'default'
    ) {}
    /**
     * Claim the next available job atomically.
     * SELECT FOR UPDATE prevents two workers from claiming the same job.
     */
    public function claimNextJob(): ?array
    {
        $this->pdo->beginTransaction();
        try {
            $stmt = $this->pdo->prepare("
                SELECT *
                FROM jobs
                WHERE queue        = :queue
                  AND status       = 'pending'
                  AND available_at <= NOW()
                ORDER BY available_at ASC
                LIMIT 1
                FOR UPDATE SKIP LOCKED
            ");
            $stmt->execute(['queue' => $this->queue]);
            $job = $stmt->fetch(\PDO::FETCH_ASSOC);
            if (!$job) {
                $this->pdo->rollBack();
                return null;
            }
            // Mark as processing - prevents other workers from claiming it
            $this->pdo->prepare("
                UPDATE jobs
                SET status      = 'processing',
                    reserved_at = NOW(),
                    attempts    = attempts + 1
                WHERE id = :id
            ")->execute(['id' => $job['id']]);
            $this->pdo->commit();
            $job['payload'] = json_decode($job['payload'], true, flags: JSON_THROW_ON_ERROR);
            return $job;
        } catch (\Throwable $e) {
            $this->pdo->rollBack();
            throw $e;
        }
    }
    /**
     * Mark a job as successfully completed.
     */
    public function markCompleted(int $jobId): void
    {
        $this->pdo->prepare("
            UPDATE jobs
            SET status       = 'completed',
                completed_at = NOW(),
                reserved_at  = NULL
            WHERE id = :id
        ")->execute(['id' => $jobId]);
    }
    /**
     * Handle a job failure - apply backoff or move to dead letter queue.
     */
    public function markFailed(int $jobId, \Throwable $exception): void
    {
        $job = $this->pdo->prepare("SELECT * FROM jobs WHERE id = ?");
        $job->execute([$jobId]);
        $job = $job->fetch(\PDO::FETCH_ASSOC);
        if (!$job) {
            return;
        }
        $errorMessage = $exception->getMessage();
        $attempts     = (int) $job['attempts'];
        $maxAttempts  = (int) $job['max_attempts'];
        if ($attempts >= $maxAttempts) {
            // Exhausted all retries - move to dead letter queue
            $this->moveToDeadLetterQueue($job, $errorMessage);
            return;
        }
        // Calculate next retry time with exponential backoff + jitter
        $availableAt = $this->calculateNextRetryTime($attempts);
        $this->pdo->prepare("
            UPDATE jobs
            SET status       = 'pending',
                available_at = :available_at,
                reserved_at  = NULL,
                last_error   = :last_error
            WHERE id = :id
        ")->execute([
            'available_at' => $availableAt,
            'last_error'   => $errorMessage,
            'id'           => $jobId,
        ]);
    }
    /**
     * Calculate the next retry timestamp using exponential backoff with jitter.
     *
     * Delay formula: BASE * 2^(attempt-1) * (1 + random jitter), capped at MAX
     *
     * Example with BASE=60s, JITTER=±25%:
     *   Attempt 1: 60s  * 1   * ~1.0  = ~60s   (45–75s)
     *   Attempt 2: 60s  * 2   * ~1.0  = ~120s  (90–150s)
     *   Attempt 3: 60s  * 4   * ~1.0  = ~240s  (180–300s)
     *   Attempt 4: 60s  * 8   * ~1.0  = ~480s  (360–600s)
     *   Attempt 5: 60s  * 16  * ~1.0  = ~960s  (720–1200s), capped at 3600s
     */
    private function calculateNextRetryTime(int $attemptNumber): string
    {
        $baseDelay    = self::BASE_DELAY_SECONDS * (2 ** ($attemptNumber - 1));
        $cappedDelay  = min($baseDelay, self::MAX_DELAY_SECONDS);
        // Apply jitter: vary delay by ±JITTER_FACTOR
        $jitterRange  = $cappedDelay * self::JITTER_FACTOR;
        $jitter       = mt_rand((int) -$jitterRange, (int) $jitterRange);
        $finalDelay   = max(1, $cappedDelay + $jitter); // minimum 1 second
        return (new \DateTimeImmutable('now', new \DateTimeZone('UTC')))
            ->modify("+{$finalDelay} seconds")
            ->format('Y-m-d H:i:s');
    }
    /**
     * Move a permanently failed job to the dead letter queue.
     */
    private function moveToDeadLetterQueue(array $job, string $failureReason): void
    {
        $this->pdo->beginTransaction();
        try {
            $this->pdo->prepare("
                INSERT INTO dead_letter_jobs
                    (original_job_id, queue, payload, failure_reason, attempts)
                VALUES
                    (:original_job_id, :queue, :payload, :failure_reason, :attempts)
            ")->execute([
                'original_job_id' => $job['id'],
                'queue'           => $job['queue'],
                'payload'         => $job['payload'],
                'failure_reason'  => $failureReason,
                'attempts'        => $job['attempts'],
            ]);
            $this->pdo->prepare("
                UPDATE jobs
                SET status    = 'dead',
                    failed_at = NOW(),
                    last_error = :last_error
                WHERE id = :id
            ")->execute([
                'last_error' => $failureReason,
                'id'         => $job['id'],
            ]);
            $this->pdo->commit();
        } catch (\Throwable $e) {
            $this->pdo->rollBack();
            throw $e;
        }
    }
    /**
     * Recover jobs that have been stuck in 'processing' too long.
     * This handles worker crashes - jobs that were claimed but never completed.
     *
     * @param int $stuckAfterMinutes Jobs reserved longer than this are considered stuck
     */
    public function recoverStuckJobs(int $stuckAfterMinutes = 30): int
    {
        $stmt = $this->pdo->prepare("
            UPDATE jobs
            SET status      = 'pending',
                reserved_at = NULL,
                available_at = NOW(),
                last_error  = 'Recovered from stuck state - worker likely crashed'
            WHERE status       = 'processing'
              AND reserved_at  < (NOW() - INTERVAL :minutes MINUTE)
        ");
        $stmt->execute(['minutes' => $stuckAfterMinutes]);
        return (int) $stmt->rowCount();
    }
}

**SKIP LOCKED** is the critical detail in the claimNextJob query. Without it, multiple workers would lock each other out — Worker A acquires a row lock, Workers B through N all block waiting for A to finish. With SKIP LOCKED, Workers B through N skip locked rows and claim the next available unlocked job. This makes the worker pool properly concurrent without lock contention.

Pattern 3: Dead-Letter Queue — Triage for Permanent Failures

Jobs that exhaust their retries end up in the dead-letter queue. This is not a failure state to be alarmed by — it is a deliberate design. The dead-letter queue separates jobs that need human attention from jobs that are processing normally. Without it, permanently failed jobs either disappear (silent data loss) or stay in the main queue (blocking healthy jobs).

The dead-letter queue requires tooling: a way to inspect failed jobs, understand why they failed, and decide what to do next (retry, discard, or fix and requeue).

<?php
final class DeadLetterManager
{
    public function __construct(private readonly \PDO $pdo) {}
    /**
     * Get unreviewed dead-letter jobs, grouped by failure reason for easier triage.
     *
     * @return array Grouped failures with counts and sample jobs
     */
    public function getPendingTriage(string $queue = 'default', int $limit = 50): array
    {
        $stmt = $this->pdo->prepare("
            SELECT
                dlj.id,
                dlj.original_job_id,
                dlj.queue,
                dlj.payload,
                dlj.failure_reason,
                dlj.attempts,
                dlj.moved_at
            FROM dead_letter_jobs dlj
            WHERE dlj.reviewed_at IS NULL
              AND dlj.queue        = :queue
            ORDER BY dlj.moved_at ASC
            LIMIT :limit
        ");
        $stmt->bindValue(':queue', $queue);
        $stmt->bindValue(':limit', $limit, \PDO::PARAM_INT);
        $stmt->execute();
        $jobs = $stmt->fetchAll(\PDO::FETCH_ASSOC);
        // Decode payloads for inspection
        foreach ($jobs as &$job) {
            $job['payload'] = json_decode($job['payload'], true);
        }
        return $jobs;
    }
    /**
     * Requeue a dead-letter job back to the main queue for another attempt.
     * Resets attempts, clears the idempotency key (to allow re-enqueueing),
     * and sets available_at to now.
     *
     * @param int    $deadLetterJobId The ID in dead_letter_jobs
     * @param string $reason          Why this job is being requeued (for the audit trail)
     */
    public function requeue(int $deadLetterJobId, string $reason): void
    {
        $dlJob = $this->pdo->prepare(
            "SELECT * FROM dead_letter_jobs WHERE id = ? FOR UPDATE"
        );
        $dlJob->execute([$deadLetterJobId]);
        $dlJob = $dlJob->fetch(\PDO::FETCH_ASSOC);
        if (!$dlJob) {
            throw new \RuntimeException("Dead-letter job {$deadLetterJobId} not found");
        }
        if ($dlJob['reviewed_at'] !== null) {
            throw new \RuntimeException("Job {$deadLetterJobId} has already been reviewed");
        }
        $this->pdo->beginTransaction();
        try {
            // Requeue into main jobs table with reset state
            // Clear idempotency_key to allow re-insertion
            $this->pdo->prepare("
                INSERT INTO jobs (queue, payload, status, attempts, max_attempts, available_at)
                VALUES (:queue, :payload, 'pending', 0, :max_attempts, NOW())
            ")->execute([
                'queue'       => $dlJob['queue'],
                'payload'     => $dlJob['payload'],
                'max_attempts' => $dlJob['attempts'] + 3, // give it more attempts
            ]);
            // Mark as reviewed with the reason
            $this->pdo->prepare("
                UPDATE dead_letter_jobs
                SET reviewed_at = NOW(),
                    resolution  = :resolution
                WHERE id = :id
            ")->execute([
                'resolution' => "Requeued: {$reason}",
                'id'         => $deadLetterJobId,
            ]);
            // Mark the original job as resolved
            $this->pdo->prepare("
                UPDATE jobs
                SET status = 'completed'
                WHERE id = :id
            ")->execute(['id' => $dlJob['original_job_id']]);
            $this->pdo->commit();
        } catch (\Throwable $e) {
            $this->pdo->rollBack();
            throw $e;
        }
    }
    /**
     * Discard a dead-letter job - mark it reviewed without requeueing.
     * Use when the job is no longer relevant (the underlying record was deleted,
     * the operation was completed by another path, etc.)
     */
    public function discard(int $deadLetterJobId, string $reason): void
    {
        $this->pdo->prepare("
            UPDATE dead_letter_jobs
            SET reviewed_at = NOW(),
                resolution  = :resolution
            WHERE id         = :id
              AND reviewed_at IS NULL
        ")->execute([
            'resolution' => "Discarded: {$reason}",
            'id'         => $deadLetterJobId,
        ]);
    }
    /**
     * Summary statistics for monitoring and alerting.
     */
    public function getStats(): array
    {
        $stats = $this->pdo->query("
            SELECT
                queue,
                COUNT(*) FILTER (WHERE status = 'pending')    AS pending,
                COUNT(*) FILTER (WHERE status = 'processing') AS processing,
                COUNT(*) FILTER (WHERE status = 'completed')  AS completed,
                COUNT(*) FILTER (WHERE status = 'dead')       AS dead
            FROM jobs
            GROUP BY queue
        ")->fetchAll(\PDO::FETCH_ASSOC);
        $dlqStats = $this->pdo->query("
            SELECT
                queue,
                COUNT(*)                                         AS total_dead,
                COUNT(*) FILTER (WHERE reviewed_at IS NULL)     AS awaiting_review,
                MIN(moved_at)                                    AS oldest_unreviewed
            FROM dead_letter_jobs
            WHERE reviewed_at IS NULL
            GROUP BY queue
        ")->fetchAll(\PDO::FETCH_ASSOC);
        return [
            'queues'      => $stats,
            'dead_letter' => $dlqStats,
        ];
    }
}

The Worker Loop: Putting It All Together

A complete worker that uses all three patterns:

<?php
// worker.php - run via: php worker.php [queue_name]
// Managed by Supervisor for process resurrection
declare(ticks=1); // Enable signal handling
$queue      = $argv[1] ?? 'default';
$pdo        = new \PDO($_ENV['DATABASE_URL'], options: [\PDO::ATTR_ERRMODE => \PDO::ERRMODE_EXCEPTION]);
$worker     = new JobWorker($pdo, $queue);
$shouldStop = false;
// Handle graceful shutdown signals from Supervisor or the OS
pcntl_signal(SIGTERM, function () use (&$shouldStop) { $shouldStop = true; });
pcntl_signal(SIGINT,  function () use (&$shouldStop) { $shouldStop = true; });
echo "Worker started on queue: {$queue}\n";
// Recover any stuck jobs from previous worker crashes on startup
$recovered = $worker->recoverStuckJobs(stuckAfterMinutes: 30);
if ($recovered > 0) {
    error_log("Recovered {$recovered} stuck jobs on startup");
}
$idleSeconds = 0;
while (!$shouldStop) {
    $job = $worker->claimNextJob();
    if ($job === null) {
        // No jobs available - sleep briefly and check again
        $idleSeconds++;
        sleep(1);
        // Log if idle for too long (helps detect queue starvation)
        if ($idleSeconds % 60 === 0) {
            error_log("Worker idle for {$idleSeconds}s on queue: {$queue}");
        }
        continue;
    }
    $idleSeconds = 0;
    $startTime   = microtime(true);
    error_log(sprintf(
        "Processing job %d: %s",
        $job['id'],
        $job['payload']['type'] ?? 'unknown'
    ));
    try {
        // Dispatch to the appropriate handler based on job type
        $handler = JobHandlerFactory::make($job['payload']['type']);
        $handler->handle($job['payload']);
        $worker->markCompleted($job['id']);
        error_log(sprintf(
            "Completed job %d in %.2fms",
            $job['id'],
            (microtime(true) - $startTime) * 1000
        ));
    } catch (\Throwable $e) {
        $worker->markFailed($job['id'], $e);
        error_log(sprintf(
            "Failed job %d (attempt %d/%d): %s",
            $job['id'],
            $job['attempts'],
            $job['max_attempts'],
            $e->getMessage()
        ));
    }
}
echo "Worker stopped gracefully.\n";

Supervisor Configuration: Process Resurrection

The worker loop above should not run as a bare cron job. It should run as a persistent process managed by Supervisor — a process control system that automatically restarts workers that crash:

; /etc/supervisor/conf.d/job-worker.conf
[program:job-worker-payments]
command                 = php /var/www/worker.php payments
directory               = /var/www
autostart               = true
autorestart             = true
startretries            = 10
startsecs               = 1
numprocs                = 3                    ; 3 parallel workers for the payments queue
process_name            = %(program_name)s_%(process_num)02d
user                    = www-data
stdout_logfile          = /var/log/supervisor/job-worker-payments.log
stdout_logfile_maxbytes = 50MB
stdout_logfile_backups  = 10
stderr_logfile          = /var/log/supervisor/job-worker-payments-error.log
[program:job-worker-default]
command                 = php /var/www/worker.php default
directory               = /var/www
autostart               = true
autorestart             = true
numprocs                = 5
process_name            = %(program_name)s_%(process_num)02d
user                    = www-data
stdout_logfile          = /var/log/supervisor/job-worker-default.log

# After creating/modifying config:
supervisorctl reread
supervisorctl update
supervisorctl start job-worker-payments:*
# Check status
supervisorctl status
# Graceful restart after code deployment
supervisorctl restart job-worker-payments:*

Supervisor gives you: automatic restart on crash, configurable worker count per queue, log rotation, and graceful shutdown on SIGTERM (which the worker loop above handles with pcntl_signal).

Monitoring and Alerting

The job queue is not finished until it is observable. At minimum, alert on these conditions:

Dead-letter queue growing: Any dead-letter jobs awaiting review mean permanently failed jobs that need human attention. Alert when awaiting_review exceeds a threshold.

Processing queue depth growing: If jobs are being enqueued faster than they are being processed, the queue depth grows. Alert when depth exceeds a threshold for more than N minutes.

Jobs stuck in processing: Workers that crashed without updating job status. The recoverStuckJobs() method handles this, but alerting when it recovers jobs is useful — it indicates a worker crash.

// Health check endpoint — returns queue statistics for monitoring systems
final class QueueHealthController
{
    public function __construct(
        private readonly JobWorker        $worker,
        private readonly DeadLetterManager $dlm
    ) {}
    public function check(): \JsonResponse
    {
        $stats = $this->dlm->getStats();
        $awaitingReview = array_sum(
            array_column($stats['dead_letter'], 'awaiting_review')
        );
        $health = $awaitingReview === 0 ? 'healthy' : 'degraded';
        return response()->json([
            'status'          => $health,
            'queues'          => $stats['queues'],
            'dead_letter'     => $stats['dead_letter'],
            'awaiting_review' => $awaitingReview,
        ], $awaitingReview > 0 ? 207 : 200);
    }
}

Wire this endpoint into your monitoring system (Datadog, Prometheus, simple uptime checker) and alert when awaiting_review > 0 or when the HTTP status is not 200.

When to Use This vs. a Managed Queue

This implementation is appropriate when:

• You need full control over the queue schema for compliance, auditing, or custom retry logic
• Your infrastructure does not include managed queue services
• You want to avoid additional operational dependencies
• The volume is manageable with a database-backed queue (typically up to several thousand jobs per minute)

Consider a managed queue service (Laravel Horizon with Redis, Amazon SQS, RabbitMQ) when:

• Volume exceeds what a database-backed queue can handle efficiently
• You need at-least-once delivery guarantees across distributed infrastructure
• Operational simplicity matters more than customization
• You need features like priority queues, delayed messages, or fan-out delivery

The patterns in this article — deduplication, backoff, and dead-letter triage — apply to managed queues too. Laravel Horizon implements all three, with Redis doing the storage instead of MySQL.

Key Takeaways

Cron is a trigger, not a job queue. Cron fires a command at a scheduled time. It does not retry failures, prevent duplicates, or give visibility into what happened. Wrap it with a system that provides these guarantees.

Idempotency keys make duplicate protection atomic. Use a UNIQUE database constraint, not application-level SELECT-then-INSERT checks. The constraint cannot be bypassed by race conditions.

Backoff with jitter prevents thundering herds. Exponential backoff spreads retry load over time. Jitter prevents synchronized retry spikes when multiple jobs fail simultaneously.

**SKIP LOCKED enables concurrent workers.** Without it, worker processes block each other. With it, each worker claims a different job without waiting for locks to release.

Dead-letter queues make failure visible and actionable. Failed jobs should not disappear silently or block the main queue. They should be isolated, triaged, and either requeued or discarded with a documented reason.

Stuck job recovery handles worker crashes. Workers that crash without completing a job leave the job in processing status indefinitely. Periodic stuck job recovery makes the queue self-healing.

Monitor queue depth and dead-letter accumulation. A growing dead-letter queue or processing backlog are the signals that something needs attention — before the silence turns into a missing payment or an undelivered report.