Background Job Monitoring in a Django SaaS: How to Know When Things Silently Fail

Background Job Monitoring in a Django SaaS: How to Know When Things Silently Fail

The Invoice That Never Sent

Three weeks. That’s how long a critical Celery task had been silently failing in one of my Django projects before anyone noticed.

The task generated monthly invoices for our SaaS customers. It ran on the 1st of each month. In April, a dependency update broke the PDF rendering. The task threw an exception, Celery retried it three times, it went to the dead letter queue, and that was it. No alert. No email. Nothing.

We found out because a customer emailed asking where their invoice was. By then, 340 invoices hadn’t been generated. The fix took 5 minutes. Finding out it was broken took 21 days.

That’s when I built a proper monitoring layer for background jobs. Not Flower — a real system that answers one question: is every job that should run actually running successfully?

Why This Matters

Background jobs in a SaaS handle the most important work — billing, notifications, data sync, report generation, webhook delivery. They run without a user watching. They fail without a user knowing.

Your web endpoints have health checks. Your API has status codes. But your background tasks? Most teams have zero visibility beyond “Celery is running.”

Celery being running and tasks completing successfully are two completely different things. A worker can be alive, consuming messages, and failing every single one. Your monitoring won’t blink.

You need three layers of visibility:

1. Task-level monitoring — did this specific task succeed or fail?
2. Heartbeat monitoring — is this periodic task still running on schedule?
3. Queue health monitoring — are tasks piling up faster than workers can process them?

Layer 1: Task-Level Success and Failure Tracking

Celery has built-in signals that fire on task success, failure, and retry. Most people never hook into them. Here’s a lightweight tracking model:

# monitoring/models.py
from django.db import models

class TaskExecution(models.Model):
    task_name = models.CharField(max_length=255, db_index=True)
    task_id = models.CharField(max_length=255, unique=True)
    status = models.CharField(max_length=20, choices=[
        ('started', 'Started'),
        ('success', 'Success'),
        ('failure', 'Failure'),
        ('retry', 'Retry'),
    ])
    started_at = models.DateTimeField(auto_now_add=True)
    completed_at = models.DateTimeField(null=True, blank=True)
    duration_ms = models.IntegerField(null=True, blank=True)
    error_message = models.TextField(blank=True)
    args = models.JSONField(default=dict, blank=True)

    class Meta:
        indexes = [
            models.Index(fields=['task_name', 'status', 'started_at']),
        ]

Now hook into Celery signals to populate it:

# monitoring/signals.py
from celery.signals import task_prerun, task_success, task_failure, task_retry
from django.utils import timezone
from .models import TaskExecution

@task_prerun.connect
def on_task_start(sender, task_id, task, args, kwargs, **kw):
    TaskExecution.objects.create(
        task_name=task.name,
        task_id=task_id,
        status='started',
        args={'args': str(args)[:500], 'kwargs': str(kwargs)[:500]},
    )

@task_success.connect
def on_task_success(sender, result, **kw):
    TaskExecution.objects.filter(task_id=sender.request.id).update(
        status='success',
        completed_at=timezone.now(),
    )

@task_failure.connect
def on_task_failure(sender, task_id, exception, traceback, **kw):
    TaskExecution.objects.filter(task_id=task_id).update(
        status='failure',
        completed_at=timezone.now(),
        error_message=str(exception)[:2000],
    )

@task_retry.connect
def on_task_retry(sender, request, reason, **kw):
    TaskExecution.objects.filter(task_id=request.id).update(
        status='retry',
        error_message=str(reason)[:2000],
    )

Register these signals in your app’s ready() method:

# monitoring/apps.py
from django.apps import AppConfig

class MonitoringConfig(AppConfig):
    name = 'monitoring'

    def ready(self):
        import monitoring.signals  # noqa

Now you have a queryable history of every task execution. But tracking isn’t alerting. Let’s add the alert layer.

Layer 2: Alerting on Failure Spikes

You don’t want an alert for every single retry. You want to know when failure rates are abnormal. Here’s a periodic check that runs every 10 minutes:

# monitoring/tasks.py
from celery import shared_task
from django.utils import timezone
from datetime import timedelta
from .models import TaskExecution

@shared_task
def check_failure_rates():
    window = timezone.now() - timedelta(minutes=30)

    # Get failure rates per task in the last 30 minutes
    from django.db.models import Count, Q

    stats = TaskExecution.objects.filter(
        started_at__gte=window
    ).values('task_name').annotate(
        total=Count('id'),
        failures=Count('id', filter=Q(status='failure')),
    )

    alerts = []
    for stat in stats:
        if stat['total'] == 0:
            continue
        failure_rate = stat['failures'] / stat['total']
        if failure_rate > 0.3:  # More than 30% failing
            alerts.append(
                f"{stat['task_name']}: {stat['failures']}/{stat['total']} "
                f"failed ({failure_rate:.0%})"
            )

    if alerts:
        send_alert('\n'.join(alerts))

For send_alert, use whatever your team already watches — Slack, PagerDuty, email:

# monitoring/alerts.py
import requests
from django.conf import settings

def send_alert(message):
    if settings.SLACK_WEBHOOK_URL:
        requests.post(settings.SLACK_WEBHOOK_URL, json={
            'text': f'⚠️ *Background Job Alert*\n```{message}```'
        })

Layer 3: Heartbeat Monitoring for Periodic Tasks

This is the layer that would have caught my invoice bug. The idea: every critical periodic task “checks in” after successful completion. If it doesn’t check in within its expected interval, fire an alert.

# monitoring/models.py (add to existing file)

class TaskHeartbeat(models.Model):
    task_name = models.CharField(max_length=255, unique=True)
    last_success = models.DateTimeField()
    expected_interval_minutes = models.IntegerField()
    is_critical = models.BooleanField(default=False)

    @property
    def is_overdue(self):
        deadline = self.last_success + timedelta(minutes=self.expected_interval_minutes)
        return timezone.now() > deadline

After each critical periodic task, update the heartbeat:

# billing/tasks.py
from celery import shared_task
from monitoring.models import TaskHeartbeat
from django.utils import timezone

@shared_task
def generate_monthly_invoices():
    # ... actual invoice logic ...

    # Check in after success
    TaskHeartbeat.objects.update_or_create(
        task_name='billing.generate_monthly_invoices',
        defaults={
            'last_success': timezone.now(),
            'expected_interval_minutes': 60 * 24 * 32,  # should run monthly
            'is_critical': True,
        }
    )

Then a watcher task checks for overdue heartbeats:

# monitoring/tasks.py (add to existing)

@shared_task
def check_heartbeats():
    overdue = TaskHeartbeat.objects.filter(is_critical=True)
    alerts = []

    for heartbeat in overdue:
        if heartbeat.is_overdue:
            hours_overdue = (
                timezone.now() - heartbeat.last_success
            ).total_seconds() / 3600
            alerts.append(
                f"{heartbeat.task_name}: last success "
                f"{hours_overdue:.1f} hours ago "
                f"(expected every {heartbeat.expected_interval_minutes} min)"
            )

    if alerts:
        send_alert('Overdue tasks:\n' + '\n'.join(alerts))

Schedule this watcher to run every hour with Celery Beat. It’s your safety net for anything periodic — nightly reports, weekly digests, monthly billing, daily data syncs.

Layer 4: Queue Depth Monitoring

Tasks piling up in a queue means workers can’t keep up. Maybe a worker died. Maybe traffic spiked. Either way, you need to know before the queue grows to 50,000 messages and tasks start timing out.

# monitoring/tasks.py (add to existing)
from django.conf import settings
import redis

@shared_task
def check_queue_depth():
    r = redis.from_url(settings.CELERY_BROKER_URL)

    queues_to_watch = {
        'celery': 1000,       # default queue, alert at 1000
        'billing': 100,       # billing queue, alert at 100
        'notifications': 500, # notifications queue, alert at 500
    }

    alerts = []
    for queue_name, threshold in queues_to_watch.items():
        depth = r.llen(queue_name)
        if depth > threshold:
            alerts.append(f"Queue '{queue_name}': {depth} pending (threshold: {threshold})")

    if alerts:
        send_alert('Queue depth warning:\n' + '\n'.join(alerts))

Putting It All Together: The Celery Beat Schedule

# config/celery.py
from celery.schedules import crontab

app.conf.beat_schedule = {
    'check-failure-rates': {
        'task': 'monitoring.tasks.check_failure_rates',
        'schedule': crontab(minute='*/10'),  # every 10 minutes
    },
    'check-heartbeats': {
        'task': 'monitoring.tasks.check_heartbeats',
        'schedule': crontab(minute='*/60'),  # every hour
    },
    'check-queue-depth': {
        'task': 'monitoring.tasks.check_queue_depth',
        'schedule': crontab(minute='*/5'),  # every 5 minutes
    },
}

Common Mistakes

1. Relying on Flower as your monitoring

Flower shows you real-time task execution. It’s a debugging tool, not a monitoring tool. It doesn’t alert you when something is missing. It shows what’s running — not what should be running but isn’t.

2. Not cleaning up old execution records

That TaskExecution table will grow fast. Add a cleanup task:

@shared_task
def cleanup_old_executions():
    cutoff = timezone.now() - timedelta(days=30)
    TaskExecution.objects.filter(started_at__lt=cutoff).delete()

3. Monitoring tasks that monitor themselves

If your monitoring relies on Celery and Celery dies, you get no alerts. Add an external watchdog — a CloudWatch alarm on queue depth, or a simple cron on a separate machine that hits a health endpoint.

4. Alerting on every failure instead of rates

Some tasks are expected to fail occasionally (external API timeouts, temporary network issues). Alert on failure rates, not individual failures. 1 failure out of 1000 is normal. 300 out of 1000 is a problem.

5. Not tracking task duration trends

A task that used to take 2 seconds and now takes 45 seconds is about to become a timeout. Track p95 duration and alert when it doubles:

# Add to task success signal
duration = (timezone.now() - execution.started_at).total_seconds() * 1000
TaskExecution.objects.filter(task_id=task_id).update(
    duration_ms=int(duration)
)

Best Practices

• Separate critical tasks into their own queue — billing tasks should never compete for workers with notification tasks
• Set explicit time limits on every task — @shared_task(time_limit=300, soft_time_limit=270) prevents zombie tasks
• Log task arguments on failure — you’ll need them to replay failed tasks
• Build an admin view — a simple Django admin page showing recent failures saves checking Slack every time
• Test your alerts — deliberately break a task in staging and confirm the alert fires. An alert you’ve never seen fire might not actually work
• Keep monitoring tasks lightweight — they should query counts, not scan millions of rows. Use the indexes defined above

Conclusion

Your background jobs are the most important code in your SaaS that nobody is watching. They handle billing, notifications, data processing — the work that keeps your business running.

The monitoring layer I’ve described here is about 200 lines of code. One model, a few signals, three periodic checks, and a Slack webhook. You can build it in an afternoon.

The alternative is finding out from a customer email three weeks later that their invoices stopped generating. I know which one I prefer.

Start with the heartbeat layer for your most critical periodic tasks. Add failure rate tracking next. Then queue depth. You’ll sleep better knowing that if something breaks at 2am, you’ll hear about it at 2:01am — not three Tuesdays later.

· · ·

Follow me for weekly Django SaaS engineering content. I write about the production problems nobody talks about in tutorials — the stuff that only matters when real customers depend on your code.