Zero Downtime

Zero-Downtime Django Deploys: Coordinating Code and Schema Changes Without a Maintenance Window

Your migration ran fine. Your tests passed. Production still went down for 90 seconds. Here’s why and how to actually fix it.

In the last article, we covered how individual migrations can lock tables and take down production. This one is about a different, sneakier problem: even when each migration is individually safe, the deploy process itself can still cause downtime.

The failure mode is always the same shape: for some window of time during a rolling deploy, old code and new code run against the same database simultaneously. If the schema and the code aren’t compatible during that window, requests fail even though both the migration and the code review looked perfectly fine in isolation.

This article is about closing that window completely.

Why Rolling Deploys Break Even “Safe” Migrations

A typical rolling deploy looks like this:

T+0s:   10 pods running old code (v1), old schema
T+0s:   Migration runs, schema changes to v2
T+5s:   Pod 1 restarts → now running new code (v2)
T+15s:  Pod 2 restarts → now running new code (v2)
...
T+90s:  Pod 10 restarts → now running new code (v2)

Between T+0s and T+90s, you have a mixed fleet: some pods on v1 code, some on v2 code, all hitting a v2 schema (or, if the migration runs after code deploy, v1 code hitting a v2 schema for some pods).

# v1 code — still running on pods 2 through 10
class Order(models.Model):
    status = models.CharField(max_length=20)

    def get_display_status(self):
        return self.status.upper()

# Migration just ran — renamed `status` to `order_status`
# v1 code above now crashes on EVERY pod that hasn't restarted yet:
# FieldError: Cannot resolve keyword 'status' into field

The fix isn’t a better migration. It’s accepting that schema changes and code changes must be decoupled across separate deploys whenever they aren’t backward-and-forward compatible.

The Core Principle: N-1 Compatibility

Every schema change must satisfy this rule:

The new schema must work with the old code, AND the old schema must work with the new code, for the duration of the rollout.

This is called N-1 compatibility, your code at version N must tolerate the schema from version N-1, and vice versa, during the overlap window.

Let’s build out the full toolkit for achieving this.

Pattern 1: Additive-Only Migrations Deploy Safely Alone

Adding a nullable column or a new table is inherently N-1 compatible old code simply ignores the new column.

# This migration is safe to deploy WITHOUT any code coordination
class Migration(migrations.Migration):
    operations = [
        migrations.AddField(
            model_name="order",
            name="fulfillment_notes",
            field=models.TextField(null=True, blank=True),
        ),
    ]

Old code doesn’t reference fulfillment_notes, so it runs unaffected. New code that uses the field only gets deployed in pods that have already restarted onto new code. No ordering constraint needed.

Rule of thumb: if git diff on your migration only contains AddField (nullable), AddIndex (concurrently), or CreateModel, you can deploy schema and code together in one release.

Pattern 2: Destructive Changes Need a 3-Release Train

Anything that removes or renames something old code depends on requires three separate releases.

Let’s walk through renaming Order.status → Order.order_status end-to-end, release by release.

Release 1: Expand, Add the New Column, Dual-Write

Migration:

# 0050_add_order_status.py
class Migration(migrations.Migration):
    operations = [
        migrations.AddField(
            model_name="order",
            name="order_status",
            field=models.CharField(max_length=20, null=True),
        ),
    ]

Code — write to both fields, read from the old one:

# models.py
class Order(models.Model):
    status = models.CharField(max_length=20)               # old — still the source of truth
    order_status = models.CharField(max_length=20, null=True)  # new — kept in sync

    def save(self, *args, **kwargs):
        self.order_status = self.status  # dual-write
        super().save(*args, **kwargs)

    def get_display_status(self):
        return self.status.upper()  # still reads old field

Backfill existing rows (run as a separate step, not in the migration):

# management/commands/backfill_order_status.py
from django.core.management.base import BaseCommand
from myapp.models import Order

class Command(BaseCommand):
    def handle(self, *args, **options):
        batch_size = 2000
        last_id = 0
        while True:
            batch = list(
                Order.objects
                .filter(order_status__isnull=True, id__gt=last_id)
                .order_by("id")
                .values_list("id", "status")[:batch_size]
            )
            if not batch:
                break

            from django.db import connection
            with connection.cursor() as cursor:
                for order_id, status in batch:
                    cursor.execute(
                        "UPDATE myapp_order SET order_status = %s WHERE id = %s",
                        [status, order_id],
                    )
            last_id = batch[-1][0]
            self.stdout.write(f"Backfilled up to id {last_id}")

Deploy Release 1. Wait for the rollout to finish completely and the backfill to complete. Do not proceed until every pod is on Release 1 and the backfill job reports done.

Release 2: Migrate Reads, Switch the Source of Truth

class Order(models.Model):
    status = models.CharField(max_length=20)               # kept, still dual-written
    order_status = models.CharField(max_length=20)          # now NOT NULL, source of truth

    def save(self, *args, **kwargs):
        self.status = self.order_status  # keep old field in sync too, for safety
        super().save(*args, **kwargs)

    def get_display_status(self):
        return self.order_status.upper()  # reads new field now

# 0051_order_status_not_null.py
class Migration(migrations.Migration):
    operations = [
        migrations.AlterField(
            model_name="order",
            name="order_status",
            field=models.CharField(max_length=20),  # null=True removed
        ),
    ]

Deploy Release 2. Wait for full rollout. At this point, no code anywhere reads status,it's only being written for safety, in case you need to roll back to Release 1.

Release 3: Contract, Remove the Old Field

Only do this once you’re confident you’ll never need to roll back past Release 2 (typically after the change has baked for at least one full deploy cycle, often a week or more in cautious teams).

class Order(models.Model):
    order_status = models.CharField(max_length=20)
    # status field removed entirely from the model

# 0052_remove_status.py
class Migration(migrations.Migration):
    operations = [
        migrations.RemoveField(
            model_name="order",
            name="status",
        ),
    ]

Three releases. Tedious, yes but at no point did old code and new schema ever disagree about what fields exist.

Pattern 3: Database-Level Defaults to Bridge the Gap

A trick that reduces the number of releases needed: use a database-level default so even unmigrated rows behave correctly the instant the column exists, removing the need for an application-level dual-write in simple cases.

# If the new field can have a sane default for ALL existing AND future rows,
# you can sometimes skip the dual-write step entirely
class Migration(migrations.Migration):
    operations = [
        migrations.AddField(
            model_name="order",
            name="priority",
            field=models.IntegerField(default=0),  # DB-level default applies to old rows too
        ),
    ]

This only works when the default is actually correct for historical data. For a rename like status → order_status, there's no universal default you need the real historical value, so dual-writing is mandatory. But for genuinely new fields (like a priority flag that didn't exist conceptually before), a default sidesteps the whole multi-release dance.

Pattern 4: Coordinating Migration Timing with Deploy Tooling

The biggest practical risk is migrations running automatically as part of the same deploy step as code rollout, with no guarantee of ordering relative to pod restarts.

The Naive (Dangerous) Approach

# DON'T do this for non-additive migrations
# deploy.yaml
steps:
  - run: python manage.py migrate
  - run: kubectl rollout restart deployment/web

If the migration includes anything destructive, pods still running old code hit the new schema mid-migration-run, and there’s no guarantee the migration finishes before traffic resumes.

Separating Migration Deploys from Code Deploys

Run migrations as an explicit, separate, observable step, not bundled silently into your CD pipeline:

# .github/workflows/deploy.yml
jobs:
  pre-deploy-migration:
    runs-on: ubuntu-latest
    steps:
      - name: Run additive migrations only
        run: |
          python manage.py migrate --plan  # review what will run
          python manage.py migrate

  deploy:
    needs: pre-deploy-migration
    runs-on: ubuntu-latest
    steps:
      - name: Rolling restart
        run: kubectl rollout status deployment/web --timeout=300s

  post-deploy-backfill:
    needs: deploy
    runs-on: ubuntu-latest
    steps:
      - name: Run data backfill (idempotent, safe to re-run)
        run: python manage.py backfill_order_status

For destructive migrations (Release 3 in the pattern above), gate them behind a manual approval step:

contract-migration:
    needs: deploy
    runs-on: ubuntu-latest
    environment:
      name: production-destructive-migration  # requires manual approval in GitHub
    steps:
      - name: Remove deprecated column
        run: python manage.py migrate myapp 0052

Pattern 5: Feature Flags to Decouple Deploy from Release

For anything riskier than a simple field rename, wrap the new code path in a feature flag so the code deploy and the behavior cutover are two independent events you control separately.

# settings.py
FEATURE_FLAGS = {
    "USE_ORDER_STATUS_FIELD": env.bool("FEATURE_USE_ORDER_STATUS_FIELD", default=False),
}

# models.py
from django.conf import settings

class Order(models.Model):
    status = models.CharField(max_length=20)
    order_status = models.CharField(max_length=20, null=True)

    def get_display_status(self):
        if settings.FEATURE_FLAGS["USE_ORDER_STATUS_FIELD"]:
            return self.order_status.upper()
        return self.status.upper()

Now the rollout sequence becomes:

1. Deploy code with the flag off, zero behavior change, fully safe
2. Once all pods are confirmed on the new code, flip the flag on via environment variable or remote config instant cutover, no deploy needed
3. If anything breaks, flip the flag back off, instant rollback, no deploy needed

This is strictly safer than relying on deploy ordering, because the cutover point becomes a config change instead of a code rollout, and config changes apply atomically across your fleet (or close to it, depending on your config propagation mechanism).

Pattern 6: Health Checks That Actually Catch Schema Drift

A subtle failure: your readiness probe says the pod is healthy, but the pod is running code incompatible with the current schema. Add a startup check that verifies schema compatibility before the pod accepts traffic.

# myapp/management/commands/check_schema_compat.py
from django.core.management.base import BaseCommand, CommandError
from django.db import connection

REQUIRED_COLUMNS = {
    "myapp_order": ["id", "order_status", "created_at"],
}

class Command(BaseCommand):
    """Run as a Kubernetes initContainer before the app container starts."""

    def handle(self, *args, **options):
        with connection.cursor() as cursor:
            for table, required_cols in REQUIRED_COLUMNS.items():
                cursor.execute(
                    "SELECT column_name FROM information_schema.columns "
                    "WHERE table_name = %s",
                    [table],
                )
                existing = {row[0] for row in cursor.fetchall()}
                missing = set(required_cols) - existing
                if missing:
                    raise CommandError(
                        f"Schema incompatible: {table} missing columns {missing}. "
                        f"Refusing to start — migration may not have run yet."
                    )
        self.stdout.write(self.style.SUCCESS("Schema compatibility check passed."))

# kubernetes deployment
spec:
  initContainers:
    - name: schema-check
      image: myapp:latest
      command: ["python", "manage.py", "check_schema_compat"]
  containers:
    - name: web
      image: myapp:latest

If a pod somehow starts before the migration has finished propagating (rare, but happens with read replicas lagging), it fails fast at startup instead of serving broken requests.

Pattern 7: Read-Replica Lag Is a Silent Killer

If you run read replicas, a migration applied to the primary doesn’t instantly exist on replicas. A pod that reads from a replica immediately after the migration runs can still see the old schema.

# DATABASE_ROUTERS setup
class ReplicaRouter:
    def db_for_read(self, model, **hints):
        return "replica"

    def db_for_write(self, model, **hints):
        return "default"

# If your migration just ran on `default`, the replica might lag by seconds.
# A read immediately after writing can hit the replica and miss the new column.

# Mitigation: force reads-after-migration to hit primary for a cooldown window
from django.db import connections

def get_order_safe(order_id, force_primary=False):
    db_alias = "default" if force_primary else "replica"
    return Order.objects.using(db_alias).get(id=order_id)

In practice, the safest approach is to treat replica propagation as part of your migration’s rollout window don’t consider a destructive migration “done” until you’ve confirmed replica lag has caught up:

# management/commands/wait_for_replica_sync.py
import time
from django.core.management.base import BaseCommand
from django.db import connections

class Command(BaseCommand):
    def handle(self, *args, **options):
        with connections["replica"].cursor() as cursor:
            while True:
                cursor.execute(
                    "SELECT EXTRACT(EPOCH FROM (now() - pg_last_xact_replay_timestamp()))"
                )
                lag_seconds = cursor.fetchone()[0] or 0
                if lag_seconds < 1:
                    self.stdout.write(self.style.SUCCESS("Replica caught up."))
                    return
                self.stdout.write(f"Replica lag: {lag_seconds:.1f}s, waiting...")
                time.sleep(2)

Run this as a gate between your migration step and your code rollout step in CI.

Putting the Full Pipeline Together

Here’s what a complete zero-downtime release looks like for a destructive schema change, end to end:

Release 1 (Expand):
  1. Deploy migration: add new nullable column
  2. Deploy code: dual-write to both columns, read from old (behind feature flag, flag OFF)
  3. Run backfill job (idempotent, batched)
  4. Wait for replica sync
  5. Verify backfill completeness with a count check

Release 2 (Migrate):
  6. Flip feature flag ON — new code path reads from new column
  7. Monitor error rates and query performance
  8. (Rollback path: flip flag OFF instantly if issues appear)

Release 3 (Contract):
  9. Deploy code: remove all references to old column, remove dual-write
  10. Wait for full rollout confirmation across all pods
  11. Deploy migration: drop old column
  12. Remove feature flag from codebase

Notice that destructive schema changes (step 11) come last, after code has been fully decoupled from the old column for at least one full release cycle. This ordering is what actually buys you zero downtime not faster migrations, not smarter locking, just strict sequencing.

Quick Reference: What Needs the Full Pipeline vs. What Doesn’t

Change Safe to deploy together? Why Add nullable field Yes Old code ignores it Add field with DB default Usually yes Old rows get default automatically Add index (CONCURRENTLY) Yes No lock, no schema dependency for old code Add new model/table Yes Old code never references it Rename field No, 3 releases Old code breaks on missing old name Remove field No verify zero references first, then 1 release Old code breaks if still referenced Change field type No 3 releases (add new, migrate, drop old) Type mismatches break serialization Add NOT NULL constraint No backfill first, then constrain Existing NULLs will violate constraint Make field unique No, verify no duplicates first Existing duplicates will fail constraint

Final Thought

Zero-downtime deploys aren’t achieved by writing faster migrations they’re achieved by never letting code and schema disagree about what exists, even for a few seconds during rollout. That means accepting that some changes take three releases instead of one, that feature flags decouple deploy from cutover, and that “the migration succeeded” is necessary but never sufficient.

The migration is the easy part. The sequencing is the actual engineering.

Next up: building a Django outbox pattern for reliable event publishing how to guarantee your database writes and your message queue never drift out of sync, even when Celery workers crash mid-task.

If you found this helpful, follow me for more Python, Django and backend deep-dives every week.

🚀 Found this helpful? 👍🏻 Like, 🔗 share, and 👉🏻 follow

Thousands of developers share what they’re building, learning, and discovering across our publications every month. One account connects you to our entire network of publications and communities. Explore more at plainenglish.io.