From Spot Chaos to Stable Multi-GPU Finetuning

What Building Resilient AI Infrastructure Taught Me About Modern ML Systems

Everyone talks about finetuning models. Very few talk about surviving reality.

The tutorials look smooth. The demos work perfectly. You launch a training run, GPUs spin up, logs start flowing, and everything feels stable — until the spot instance disappears halfway through training, checkpoints fail to sync, dependencies drift, environments corrupt themselves, and hours of compute vanish instantly.

That’s the side of AI infrastructure most people never see.

Over the last few weeks, I explored a deeply engineered workflow for resilient spot GPU finetuning — a system designed not just to train models, but to survive interruptions, recover state safely, and behave like fault-tolerant infrastructure instead of a fragile experiment. What fascinated me wasn’t the model architecture itself. It was the operational discipline behind keeping unreliable spot infrastructure alive long enough to produce stable results.

The deeper I looked, the clearer one truth became:

Spot training is an operations problem first, and a modeling problem second.

And honestly, that sentence completely changed how I think about modern AI engineering.

When the Real Problem Isn’t the Model

The instinctive reaction when training fails is to blame the model:

• bad hyperparameters
• wrong learning rate
• optimizer instability
• dataset issues

But in distributed spot infrastructure, failures often come from somewhere else entirely:

• interrupted instances
• broken dependencies
• concurrency corruption
• failed environment setup
• unsynchronized checkpoints
• orchestration race conditions

In other words: the problem shifts from machine learning into distributed systems engineering.

One of the most interesting things about this architecture was how openly it embraced this reality. The workflow wasn’t pretending spot interruptions wouldn’t happen. Instead, it accepted instability as a fundamental property of cheap compute and engineered around it.

That mindset is incredibly important because modern AI systems are increasingly becoming operational systems, not just research experiments.

The infrastructure I studied used Verda’s spot GPU platform with Axolotl, distributed PyTorch, Hugging Face synchronization, and resilient orchestration to finetune a Qwen2.5–14B model across multiple RTX 6000 Ada GPUs.

What stood out wasn’t simply the hardware. It was the architecture around survivability.

The system focused on four operational guarantees:

• keep compute costs low
• survive interruptions
• continuously sync checkpoints
• safely terminate infrastructure only after persistence completes

That is a very different philosophy from “just launch training.”

The Three-Pillar Architecture

One of the cleanest design decisions in the workflow was the separation of responsibilities into three distinct layers.

Instead of building a monolithic launcher script, the system divided itself into:

1. the control plane
2. the execution plane
3. the persistence plane

That separation ended up becoming the backbone of resilience.

1. Local Orchestrator — The Control Plane

The local orchestrator acts as the command center.

It provisions spot VMs on Verda, attaches SSH keys, uploads launch bundles, monitors status markers, and polls execution state. But the most important design choice wasn’t what it did — it was what it was not allowed to do.

The orchestrator was intentionally restricted from deleting infrastructure until the remote system explicitly confirmed:

• training completion
• final checkpoint sync
• terminal status writes

This matters more than it sounds.

Because in spot infrastructure, teardown before persistence equals total data loss.

That single rule transforms unreliable hardware into recoverable infrastructure.

2. Remote Launcher — The Data Plane

The remote launcher handled:

• dependency installation
• environment bootstrapping
• distributed execution
• runtime synchronization
• process locking

This layer was heavily defensive by design.

One particularly smart feature was the use of strict lock files and .venv integrity checks to prevent duplicate launchers from corrupting training state.

This is the kind of problem that rarely appears in toy tutorials but becomes critical in real distributed systems. Concurrency bugs in spot infrastructure don’t just slow training down — they can invalidate entire runs.

The workflow also aggressively pinned dependencies:

• torch versions
• torchvision versions
• CUDA-related libraries
• runtime environments

That may sound boring, but dependency drift quietly destroys more training runs than most model bugs ever will.

One of the most valuable lessons from this architecture was that resilience often comes from boring engineering discipline:

• idempotency
• deterministic environments
• defensive execution ordering
• strict synchronization contracts

Not glamorous AI breakthroughs.

Just solid systems engineering.

Checkpoint Safety Became the Entire Game

The more I explored the workflow, the more it became obvious that checkpoint synchronization was the real heart of the system.

Everything revolved around preserving state.

The architecture enforced a strict sequence:

1. generate final checkpoint
2. sync logs and artifacts
3. upload to Hugging Face
4. write terminal markers
5. only then permit teardown

And the document described this sequence as:

“non-negotiable.”

Honestly, that word fits perfectly.

Because spot infrastructure changes your relationship with persistence completely. On local hardware, saving checkpoints feels routine. On preemptible infrastructure, checkpoints become survival boundaries.

A VM disappearing is no longer catastrophic if state already escaped the machine.

That’s the real objective: not preventing failure, but making failure survivable.

This mindset felt surprisingly similar to operating system design.

Operating systems assume:

• processes crash
• hardware fails
• interruptions happen

Their job is not eliminating failure — it’s managing recovery safely.

Modern AI infrastructure is starting to inherit the exact same philosophy.

The Compute vs Memory Illusion

One section exposed a misconception I see constantly in AI discussions: people often assume low VRAM usage means GPUs are idle.

But the QLoRA workflow in this system was actually compute-bound, not memory-bound.

The GPUs showed:

• relatively low VRAM usage (~18–19GB)
• but extremely high compute saturation (~95–98%)

That distinction is incredibly important.

Modern finetuning performance depends less on “maxing out VRAM” and more on:

• kernel utilization
• throughput efficiency
• synchronization overhead
• compute saturation
• pipeline orchestration

This is one reason why simplistic “VRAM calculators” often fail to predict real-world performance.

The infrastructure layer matters just as much as the model itself.

And honestly, this section highlighted something bigger: AI engineering is increasingly becoming systems engineering.

We are no longer just training models. We are designing:

• schedulers
• memory systems
• synchronization layers
• orchestration pipelines
• persistence guarantees

The ML stack is quietly evolving into distributed operating systems.

Failure Wasn’t Avoided — It Was Engineered Around

What I appreciated most about the workflow was its honesty.

The architecture never assumed ideal conditions. It assumed:

• spot interruptions
• partial failures
• corrupted environments
• network instability
• inconsistent external APIs

And then it designed recovery mechanisms around those assumptions.

That philosophy appeared everywhere:

• aggressive dependency pinning
• lock files
• forced environment regeneration
• disabled optional telemetry
• defensive bootstrap sequencing
• periodic checkpoint synchronization

This is what mature infrastructure engineering looks like.

Not removing failure. Containing it.

Why Verda’s Spot Infrastructure Matters

One reason this workflow was even feasible was access to affordable spot GPU infrastructure from Verda.

The entire system depended on:

• rapid provisioning
• scalable GPU availability
• distributed orchestration
• cost-efficient experimentation

Without affordable spot compute, repeatedly testing:

• failure recovery
• synchronization guarantees
• checkpoint survivability
• distributed launch flows

would become prohibitively expensive very quickly.

That’s what makes spot infrastructure so interesting: it forces better engineering discipline while dramatically lowering experimentation cost.

And when paired with resilient orchestration, unreliable hardware becomes surprisingly powerful.

The Five Rules That Stayed With Me

The workflow concluded with five operational rules for resilient spot finetuning, and honestly, they summarize modern AI infrastructure engineering beautifully:

1. Spot training is an operations problem first.
2. Dependency drift kills more runs than model bugs.
3. Locking and idempotency are mandatory for survival.
4. Final checkpoint synchronization before teardown is non-negotiable.
5. Scale through controlled smoke tests before full runs.

These lessons extend far beyond finetuning.

They apply to almost every distributed AI system being built today.

Final Thoughts

What fascinated me most about this architecture wasn’t the model size or the GPU count.

It was the realization that modern AI systems are quietly becoming operating systems.

They:

• provision resources
• recover state
• synchronize memory
• coordinate execution
• survive failure
• manage persistence across unreliable infrastructure

The future of AI engineering may depend less on inventing bigger models and more on building smarter systems around them.

And somewhere in that transition, spot chaos slowly becomes repeatable infrastructure.