A GitOps Pattern for Database Migrations and Application Deployments on Kubernetes

Intro

GitOps works well for deploying applications, but database schema migrations can still be a tricky problem.

In the Kubernetes environments I have worked with, application releases are managed through GitOps. However, schema migrations are often executed separately using CI pipelines such as GitHub Actions that apply Kubernetes Jobs directly.

Problem Statement

This approach has a limitation. From the perspective of a platform team, it is desirable to introduce guardrails to prevent applications from being updated before the database schema migration has completed.

In a GitOps-managed platform where schema migrations are executed outside of GitOps, establishing a dependency between application releases and database schema migrations is not straightforward.

This led me to explore whether GitOps tooling could be used to introduce an explicit dependency between database migrations and application releases.

Preliminary Research

To explore whether this dependency could be expressed within GitOps itself, I started by investigating the capabilities of popular GitOps tools such as ArgoCD and FluxCD.

ArgoCD: During my research, I came across a document titled **Sync Phases and Waves** in the ArgoCD documentation.

This document describes how dependencies between Kubernetes resources such as Pods and Jobs can be controlled using sync phases and waves.

FluxCD: During my research, I found an article titled **Running pre and post-deployment jobs with Flux**

This document describes how to use dependsOn + Kustomization resource to define dependencies between jobs and deployments in FluxCD.

Since both ArgoCD and FluxCD appear to support mechanisms for expressing such dependencies, I decided to test these approaches using both tools.

Preliminary Validation

Below is a simplified example configuration.(This assumes that operator and other necessary components are already installed.)

ArgoCD(Sync Phases and Waves):

apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
  name: sample-app
  namespace: argocd
  finalizers:
    - resources-finalizer.argocd.argoproj.io
spec:
  project: default
  source:
    repoURL: https://github.com/<sample-repo>
    targetRevision: HEAD
    path: ./gitops/
  destination:
    server: https://kubernetes.default.svc
    namespace: default
  syncPolicy:
    automated:
      prune: true
      selfHeal: true
    syncOptions:
      - CreateNamespace=true

apiVersion: batch/v1
kind: Job
metadata:
  name: db-migration
  annotations:
    # Run before the synchronization starts
    argocd.argoproj.io/hook: PreSync
    # Delete the previous job before creating a new one
    argocd.argoproj.io/hook-delete-policy: BeforeHookCreation
    # Execution order (lower numbers run first)
    argocd.argoproj.io/sync-wave: "1"
spec:
  template:
    spec:
      containers:
      - name: migration
        image: alpine:latest
        command: ["/bin/sh", "-c"]
        args: ["echo 'Running database migrations...'; sleep 10; echo 'Done!'"]
      restartPolicy: Never
  backoffLimit: 1
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: sample-app
  labels:
    app: sample-app
  annotations:
    # Run after the migration job (Wave 1) completes successfully
    argocd.argoproj.io/sync-wave: "2"
spec:
  replicas: 2
  selector:
    matchLabels:
      app: sample-app
  template:
    metadata:
      labels:
        app: sample-app
    spec:
      containers:
      - name: nginx
        image: nginx:latest
        ports:
        - containerPort: 80
---
apiVersion: v1
kind: Service
metadata:
  name: sample-app-service
spec:
  selector:
    app: sample-app
  ports:
    - protocol: TCP
      port: 80
      targetPort: 80
  type: ClusterIP

We deploy resources like the ones mentioned above to ArgoCD.

Although the diagram does not explicitly show the ordering, running the synchronization revealed that the Deployment was updated only after the Job had completed.

The key point is that in ArgoCD, a PreSync Job is not continuously reconciled as part of the desired state. Instead, it is executed as a hook.

Because of this behavior, even if the Job is deleted after execution, it will be triggered again before the Deployment is applied. This ensures that the migration runs before the application update.

This mechanism allows the migration job to be coordinated with the application deployment through the GitOps workflow, helping address the initial problem.

FluxCD( **Running pre and post-deployment jobs with Flux*):***

.
├── flux-system
│   └── app.yaml
├── gitops_flux
│   ├── deployment
│   │   └── deployment.yaml
│   └── job
│       └── job.yaml

# flux-system/app.yaml
apiVersion: source.toolkit.fluxcd.io/v1
kind: GitRepository
metadata:
  name: my-app-repo
  namespace: flux-system
spec:
  interval: 1m0s
  ref:
    branch: main
  url: https://github.com/your-username/your-repo.git
  secretRef:
    name: flux-github-token
---
apiVersion: kustomize.toolkit.fluxcd.io/v1
kind: Kustomization
metadata:
  name: pre-deploy
  namespace: flux-system
spec:
  interval: 10m
  path: ./gitops_flux/job
  prune: true
  wait: true # Wait for the Job to complete successfully
  force: true # Recreate the Job when the spec (e.g., image) changes
  sourceRef:
    kind: GitRepository
    name: flux-system
---
apiVersion: kustomize.toolkit.fluxcd.io/v1
kind: Kustomization
metadata:
  name: deploy
  namespace: flux-system
spec:
  dependsOn:
    - name: pre-deploy # This is the "PreSync" equivalent
  interval: 10m
  path: ./gitops_flux/deployment
  prune: true
  wait: true
  sourceRef:
    kind: GitRepository
    name: flux-systemgitops_flux

# gitops_flux/deployment/deployment.yaml
apiVersion: apps/v1
kind: Deployment
metadata:
  name: sample-app-flux
  namespace: default
  labels:
    app: sample-app-flux
spec:
  replicas: 2
  selector:
    matchLabels:
      app: sample-app-flux
  template:
    metadata:
      labels:
        app: sample-app-flux
    spec:
      containers:
      - name: nginx
        image: nginx:latest
        ports:
        - containerPort: 80
        env:
        - name: release_version
          value: "2"
---
apiVersion: v1
kind: Service
metadata:
  name: sample-app-flux-service
  namespace: default
spec:
  selector:
    app: sample-app-flux
  ports:
    - protocol: TCP
      port: 80
      targetPort: 80
  type: ClusterIP

# gitops_flux/job/job.yaml
apiVersion: batch/v1
kind: Job
metadata:
  name: db-migration-flux
  namespace: default
spec:
  template:
    spec:
      containers:
      - name: migration
        image: alpine:latest
        command: ["/bin/sh", "-c"]
        args: ["echo 'Running database migrations...'; sleep 10; echo 'Done!'"]
      restartPolicy: Never
  backoffLimit: 1

We deploy resources like the ones mentioned above to FluxCD.

Although the diagram does not explicitly show the dependency, running the synchronization revealed that the Deployment was updated only after the Job completed.

In ArgoCD, a PreSync resource is executed as a hook. In contrast, FluxCD uses dependsOn and Kustomization to express dependencies, which means the Job itself becomes part of the GitOps-managed resources.

This raises an interesting question. If a Pod associated with a Job is deleted, the Job still exists, so from the perspective of Kustomization it should remain in a ready state. But what happens if the Job itself is deleted? Let’s try deleting it to see.

kubectl delete job db-migration-flux -n default

When the Job was deleted, the migration started running again. In other words, when a dependency is defined between a Job and a Deployment in Flux, deleting the Job may trigger the Job to run again.

This means the migration could be executed at an unintended time.

Once the Job runs again and completes successfully, it returns to the “Ready” state.

 ❯ kubectl get job,pods
NAME                          STATUS     COMPLETIONS   DURATION   AGE
job.batch/db-migration-flux   Complete   1/1           3m24s      4m53s

NAME                                   READY   STATUS      RESTARTS   AGE
pod/db-migration-flux-lx69d            0/1     Completed   0          4m53s
pod/sample-app-flux-68b6bfc5bb-4nmx9   1/1     Running     0          15m
pod/sample-app-flux-68b6bfc5bb-xnsk8   1/1     Running     0          15m

Limitations of Job-based Approaches in GitOps

With ArgoCD, Jobs defined as hooks are not treated as part of the desired state managed by GitOps. As a result, deleting the Job after execution generally does not cause issues, because the hook will simply run again during the next synchronization.

In contrast, FluxCD manages Jobs as regular GitOps resources through Kustomization. This means that deleting a Job triggers reconciliation, which may cause the Job to run again unintentionally.

In other words, while the ArgoCD approach fits naturally with hook-based workflows, the FluxCD approach requires additional care to ensure that Jobs are not deleted unintentionally.

Rethinking Migration Management in GitOps

To address the issue where unintended reconciliation causes Jobs to be re-created when using FluxCD, as described in the previous section, I implemented a small custom operator that introduces a custom resource called InitJob

[embed]GitHub - S-mishina/initjob-operator Contribute to S-mishina/initjob-operator development by creating an account on GitHub.github.com

The mechanism itself is relatively simple.

I introduced a custom resource called InitJob as a higher-level abstraction over the Kubernetes Job resource. The operator manages the lifecycle of Jobs through this resource.

This approach helps address the problem described earlier because Kubernetes Jobs are typically used for short-lived tasks rather than long-lived desired state.

By introducing a custom resource that maintains persistent state, the system can preserve the intended state even if the Job or its Pods are deleted. As a result, the Kustomization state remains consistent.

Let’s take a look.

Example: InitJob Resource

.
├── flux-system
│   └── app1.yaml
├── gitops_flux
│   ├── deployment1
│   │   └── deployment.yaml
│   ├── initjob
│   │   └── init_job.yaml

# flux-system/app1.yaml
apiVersion: source.toolkit.fluxcd.io/v1
kind: GitRepository
metadata:
  name: my-app-repo
  namespace: flux-system
spec:
  interval: 1m0s
  ref:
    branch: main
  url: https://github.com/S-mishina/gitops-database-migration-sandbox.git
  secretRef:
    name: flux-github-token
---
apiVersion: kustomize.toolkit.fluxcd.io/v1
kind: Kustomization
metadata:
  name: pre-deploy-initjob
  namespace: flux-system
spec:
  interval: 5s
  path: ./gitops_flux/initjob
  prune: true
  wait: true # Wait for the InitJob to be healthy
  force: true
  sourceRef:
    kind: GitRepository
    name: my-app-repo
---
apiVersion: kustomize.toolkit.fluxcd.io/v1
kind: Kustomization
metadata:
  name: deploy-initjob
  namespace: flux-system
spec:
  dependsOn:
    - name: pre-deploy-initjob # This is the "PreSync" equivalent
  interval: 5s
  path: ./gitops_flux/deployment1
  prune: true
  wait: true
  sourceRef:
    kind: GitRepository
    name: my-app-repogitops_flux

# gitops_flux/deployment1/deployment.yaml
apiVersion: apps/v1
kind: Deployment
metadata:
  name: sample-app-flux
  namespace: default
  labels:
    app: sample-app-flux
spec:
  replicas: 2
  selector:
    matchLabels:
      app: sample-app-flux
  template:
    metadata:
      labels:
        app: sample-app-flux
    spec:
      containers:
      - name: nginx
        image: nginx:latest
        ports:
        - containerPort: 80
        env:
        - name: release_version
          value: "8"
---
apiVersion: v1
kind: Service
metadata:
  name: sample-app-flux-service
  namespace: default
spec:
  selector:
    app: sample-app-flux
  ports:
    - protocol: TCP
      port: 80
      targetPort: 80
  type: Cluster

# gitops_flux/inijob/job.yaml
apiVersion: batch.init.sre.ryu-tech.blog/v1alpha1
kind: InitJob
metadata:
  name: sample-initjob
  namespace: default
spec:
  jobTemplate:
    spec:
      ttlSecondsAfterFinished: 0
      template:
        spec:
          containers:
          - name: migration
            image: alpine:latest
            command: ["/bin/sh", "-c"]
            args: ["echo 'Running database migrations...'; sleep 10; echo 'Done!'"]
          restartPolicy: Never
      backoffLimit: 0

 ❯ kubectl get initjob,job,pods
NAME                                                  PHASE       JOB                       SUCCEEDED   AGE
initjob.batch.init.sre.ryu-tech.blog/sample-initjob   Succeeded   sample-initjob-b09351cb   true        7m37s

NAME                                           READY   STATUS      RESTARTS   AGE
pod/sample-app-flux-init-job-78ff6fdcf-49p9g   1/1     Running     0          49s
pod/sample-app-flux-init-job-78ff6fdcf-4t9s8   1/1     Running     0          52

As mentioned earlier, by using InitJob, the Flux Kustomization remains in the Ready state even after the Job is deleted, since the InitJob resource persists.

Conclusion

In this article, we explored how database schema migrations can be coordinated with application deployments in a GitOps-based platform.

Through experiments with both ArgoCD and FluxCD, we identified two practical approaches for introducing dependencies between database migrations and application releases.

ArgoCD

Dependencies between Jobs and Deployments can be established using hook mechanisms such as PreSync hooks combined with sync waves.

FluxCD

Dependencies can be expressed declaratively using Kustomization and dependsOn relationships.

However, our experiments also suggested a difference between the two tools. In ArgoCD, Jobs executed as hooks are treated as part of the synchronization workflow rather than as long-lived desired state. In contrast, FluxCD manages Jobs as regular GitOps resources through reconciliation.

Because Kubernetes Jobs are inherently one-shot resources, this behavior can sometimes introduce challenges in FluxCD environments. For example, when a Job is deleted, reconciliation may recreate it and trigger the migration again, which could lead to unintended executions.

To explore a possible approach, we implemented a small custom operator called InitJob, which introduces a higher-level resource that manages the lifecycle of migration Jobs. In our experiments, this approach appeared to work because the custom resource maintains persistent state even when the underlying Job is removed.

At the same time, introducing and maintaining a custom operator may add operational overhead. This also points to a broader design challenge: handling one-shot workflows such as database migrations within a reconciliation-based GitOps model is not always straightforward.

One possible takeaway from this exploration is the contrast in design philosophy between ArgoCD and FluxCD. ArgoCD provides hooks that allow workflow-like behavior during synchronization, while FluxCD focuses on managing resources through declarative reconciliation. Both approaches have advantages, but they also reflect different trade-offs in how GitOps systems handle operational workflows.

Finally, if you are interested in this approach, we’d be really happy to hear your feedback — for example, by opening an issue in the GitHub repository.