GitOps for ML in 2026: Treat Your AI Models Like Microservices (Or Watch Them Drift Into Production Chaos)

The fraud detection model had been serving wrong predictions for six hours when the first complaint came in.

A customer called to say the platform had blocked a legitimate transaction. Then another. The support team escalated. An engineer pulled up the monitoring dashboard and saw normal latency, normal throughput, no errors. Everything looked healthy. The model was serving. It just wasn’t serving correctly.

It took another two hours to figure out what happened. Someone on the data team had modified the feature pipeline upstream, changing how one of the categorical inputs was encoded. The model trained on the old encoding was now receiving inputs in a different format. It was still producing predictions, just increasingly bad ones.

The fix was straightforward once you understood it. Rolling it back was not. There was no rollback. The model had been deployed by a platform engineer who SSHed into the serving cluster, updated a config file, and restarted the service. That engineer was now in a different timezone, unreachable. There was no record of what version had been running before, no artifact stored with the previous weights, no way to flip a switch and go back.

The team spent three hours retraining an older version from a checkpoint, testing it, and redeploying it manually. Total impact: eight hours of degraded predictions, two days of incident review, and a very tense engineering retrospective.

That incident is not unusual. A version of it plays out at companies large and small every time ML model deployments are treated as one-off operations rather than managed software artifacts.

The Insight Nobody Wants to Hear

Software engineers figured out how to manage application deployments reliably about a decade ago. The tools are called Git, CI/CD, and infrastructure-as-code. Every application change goes through a pull request. Every deployment is triggered by a merge to a specific branch. Every environment is described in a file that lives in version control. Rolling back means reverting a commit.

ML teams looked at all of this and largely decided it didn’t apply to them. Models are different, the thinking went. They’re not code, exactly. They’re artifacts produced by a training run. You can’t put a 2GB model file in Git. The versioning lives in the experiment tracker.

All of that is true and none of it is an excuse for the workflow gap that results.

The insight is this: you don’t put the model weights in Git. You put the reference to the model weights in Git. A single line in a YAML file says “this environment is running model version 47, which lives at this S3 path.” When that line changes, via a pull request, with a reviewer, with a comment explaining why, a deployment happens automatically. When it needs to roll back, you revert the commit and the deployment happens in reverse.

This is GitOps. Teams have been applying it to application workloads for years. Applying it to ML model serving is not a breakthrough, it’s just overdue.

What the System Actually Looks Like

The architecture has three main pieces working together.

MLflow handles experiment tracking and model registration. Every training run produces a version number and stores the artifact in S3. The model registry gives you stage management (Staging, Production) and a clean API for querying which version is which. MLflow solves the “where did this model come from” question reliably. It does not solve the “how is this model deployed” question.

KServe handles the Kubernetes serving layer. Instead of managing a deployment and a service and a scaling policy by hand, you define an InferenceService custom resource. You give it an S3 path, a resource request, and a replica count. KServe handles downloading the artifact, loading it into the right serving framework, exposing an HTTP endpoint, and autoscaling under load. It also supports canary deployments natively: you can tell it to route 10% of traffic to a new version while 90% continues to the stable version, without touching Istio config manually.

ArgoCD handles the reconciliation loop. You point it at a Git repository containing your InferenceService definitions. It watches for changes. When a commit modifies the model version in a values file, ArgoCD applies the change to the cluster within minutes. If someone manually edits the InferenceService directly on the cluster (old habits die hard), ArgoCD detects the drift and reverts it. The cluster always reflects what’s in Git, not what someone did at 2am last Tuesday.

The Cultural Shift Is the Hard Part

The technology is straightforward. The harder part is getting a team to agree that Git is the source of truth for model deployments, full stop.

That means data scientists don’t deploy models by updating the MLflow stage to “Production” and assuming something will pick it up. It means platform engineers don’t push model updates by SSHing into serving nodes. It means the question “what version is running in production right now?” has a one-second answer: look at the values.yaml file in the prod environment directory.

The promotion workflow that makes this real is simple. When a data scientist registers a new model version in MLflow, a CI pipeline opens a pull request bumping the version number in the dev environment’s values file. The data scientist or a reviewer approves and merges it. ArgoCD deploys to dev automatically. After validation, a second pull request promotes to staging. After the staging soak period (typically 24 to 48 hours with automated evaluation running), a third pull request promotes to prod with the canary flag enabled.

That last step deploys the new version at 10% traffic. Prometheus monitors latency and prediction distribution. If both look healthy after two hours, an automated pipeline opens the full-cutover PR. If they don’t, the engineer runs git revert HEAD~1, and the previous version is back in four minutes.

Before this system, a production model deployment took 4 to 6 hours of engineer time. The new workflow brings it to 8 minutes from model registration to production canary. The rest of the time is soak: intentional, monitored, automatically resolved.

What Drift Looks Like When You’re Watching

The incident that opened this article would have gone differently with proper monitoring in place.

KServe’s serving framework exposes prediction distributions as Prometheus metrics. You can write alerting rules that fire when the distribution of predictions shifts significantly from a recent baseline. A schema change in the feature pipeline shows up as a spike in this metric within minutes, not hours, because you’re watching the model’s behavior continuously, not waiting for a customer to notice something is wrong.

In practice, a data schema change that previously went undetected for six hours was caught in 15 minutes on the new system. The alert fired, the on-call engineer saw that a canary was active, ran the revert, and the rollback completed in 4 minutes. No customer impact beyond the canary population, which was 10% of traffic.

That’s not a marginal improvement. That’s the difference between an incident that affects a small slice of traffic for 20 minutes versus one that affects all traffic for eight hours.

Three rollbacks were executed in the first month after this system was deployed. The average time from decision to stable serving was 4 minutes each. The team started calling it “boring deployment week” because it felt like nothing was happening, which was exactly the point.

The Audit Trail You Never Had

There’s a secondary benefit that doesn’t show up in incident metrics but matters enormously for regulated industries.

Every model promotion is a pull request. The PR contains the version number, the S3 artifact path, and whatever context the data scientist included in the description. The reviewer’s approval is recorded. The merge timestamp is recorded. Git history shows you exactly what was running in production on any given day, who approved it, and what the stated justification was.

This is table stakes for application code. For ML models, it’s still rare enough that it surprises people when they see it working.

The question “who deployed this?” went from “check Slack history and hope” to “git log, done.”

Starting This at Your Team

The hardest part of starting is getting agreement that Git owns the desired state for model serving, not MLflow stages, not a configuration database, not a spreadsheet.

Once you have that agreement, start small. Pick one model. Set up KServe for that model in dev. Write the ArgoCD Application pointing at a values.yaml with the model version. Make one promotion work end to end before adding staging and prod.

Add the Prometheus alerting after the basic workflow is solid. The drift detection and latency alerts are what make the system trustworthy enough to use in production. Without them, you’re flying blind on a shorter deployment cycle, which is not actually better.

The canary workflow is the last piece. It’s optional for models with low traffic volume, where the statistical signal from a 10% canary would take days to accumulate. For high-volume models, it’s the most important safety mechanism in the stack.

The teams that implement this fully tend to describe the same experience: the first few weeks feel like overhead, because you’re building discipline around something that used to be ad hoc. After that, the discipline pays compound interest. Deployments become events that happen, rather than events that are survived.

The fraud detection team never had another eight-hour incident. Not because the models stopped having problems, but because when problems happened, they caught them in 15 minutes and resolved them in 4.

If you’re running into this at your team, I’d be curious what the biggest sticking point has been. The technical setup is usually the easy part. The organizational piece tends to be where the real work lives.

Clap if this was useful. Follow for more on platform engineering and ML infrastructure.