How I Migrated 200+ VMware VMs to Kubernetes-Native Infrastructure-Without a Single Weekend Outage

There’s a particular kind of silence in a war room at 2:47 AM when 200 production virtual machines are mid-flight between two fundamentally different infrastructure paradigms. Not the silence of calm — the silence of 14 engineers collectively holding their breath.

That was March of last year. A mid-sized financial services firm, 2,400 employees, $3.8B AUM under management platform. Their entire trading analytics stack — pricing feeds, risk aggregation services, compliance archival — running on a brownfield vSphere 6.7 environment that had accumulated six years of technical debt, unlabeled snapshots, and an NSX-V deployment that VMware had already sunset.

This is the story of how we got them out of that, cleanly, onto VMware Cloud Foundation 9 with VKS-enabled workloads — and what I learned about the gap between what architecture diagrams promise and what production infrastructure actually does.

The Engagement Brief (And What It Didn’t Tell Me)

The initial SOW read simply: “Migrate legacy VMware environment to modernized platform. Enable Kubernetes for new application workloads.”

Clean. Concise. Wildly optimistic.

When I joined the engagement, I spent the first three days doing nothing but listening. Sat in on the ops team’s morning standup. Watched how the infrastructure team handled a storage alert on one of their vSAN clusters. Read the last eighteen months of change request tickets.

What I found:

• Four separate vCenter instances, originally consolidated “temporarily” in 2021, now permanent. No linked-mode. No unified RBAC model.
• NSX-V for microsegmentation — end-of-life since January 2022, still managing 60+ security groups across production workloads.
• vSAN stretched cluster across two datacenter-adjacent colo cages, running 6.7 U3, with a witness appliance that had last been patched in 2020.
• 48 VMs with manual IP assignments baked into guest OS configs — no IPAM, no DHCP, just spreadsheets and institutional memory.
• Compliance constraints: SOC 2 Type II audit in five months. Any migration that touched the compliance boundary required a change advisory board (CAB) review with 10-business-day lead time.

None of this was in the SOW.

This is the first real lesson of enterprise infrastructure work: the document describes what the client thinks exists. The actual environment is what they’ve forgotten to document.

Designing the Target State

Before touching a single VM, I spent two weeks on architecture design and stakeholder alignment. This sounds slow. It isn’t. A bad migration plan executed quickly is how you end up rebuilding your resume.

The Target: VCF 9 + VKS

The target architecture was:

• VMware Cloud Foundation 9 as the unified SDDC platform — vSphere 8, vSAN 8 Express Storage Architecture (ESA), NSX 4.x replacing the end-of-life NSX-V
• VMware vSphere Kubernetes Service (VKS) to enable Tanzu Kubernetes Clusters for net-new application workloads
• Supervisor Clusters on three vSphere namespaces: prod, staging, platform
• Forklift Operator (from the KubeVirt ecosystem, deployed via Alauda Container Platform) for the actual VM migration pipeline

The client had already standardized on Alauda Container Platform for their Kubernetes management layer — a decision made before I joined. My job was to integrate the migration tooling into that existing control plane rather than introduce new management surface area.

Phasing: The Non-Negotiable

Given the SOC 2 timeline, compliance workloads had to move last. That shaped everything:

Phase 1 - Infrastructure Foundation    (Weeks 1–4)
  VCF 9 deployment, NSX-V → NSX 4 migration
  vSAN ESA configuration, HA/DR policies

Phase 2 - Pilot Migration              (Weeks 5–7)
  20 non-production VMs via Forklift
  Network and storage mapping validation
  Runbook refinement

Phase 3 - Wave 1 Production            (Weeks 8–11)
  ~130 VMs: dev tooling, analytics, internal apps
  Forklift warm migration with cutover windows

Phase 4 - Compliance Boundary          (Weeks 12–16)
  ~50 VMs: trading platform, archival, risk systems
  CAB-approved change windows
  Parallel-run validation before cutover

The NSX-V to NSX 4 Problem

This was the hardest part of the project, and it’s the part that most architecture docs gloss over.

NSX-V security groups don’t have a direct migration path to NSX-T/4.x. The data models are fundamentally different. NSX-V works with vSphere distributed port groups and vCenter inventory. NSX-T works with its own logical overlay model.

We had 60+ security groups, some of them nested three levels deep. The client’s security team had built these over five years and frankly, no one had a complete mental model of them.

What I did:

Step 1: Export and audit. I wrote a PowerCLI script to export every NSX-V security group, its members, and every firewall rule referencing it into a structured JSON. Then I ran a second pass to map VM membership into those groups using vSphere tags as a proxy.

$secGroups = Get-NsxSecurityGroup
foreach ($sg in $secGroups) {
    $members = Get-NsxSecurityGroupEffectiveMember -SecurityGroup $sg
    [PSCustomObject]@{
        Name    = $sg.name
        Members = $members.IPSet.value -join ","
        Rules   = (Get-NsxFirewallRule | Where-Object {$_.sources.source.value -eq $sg.objectId -or $_.destinations.destination.value -eq $sg.objectId}).count
    }
} | Export-Csv nsxv-audit.csv

Step 2: Rationalize. Of 60 security groups, 22 had zero active firewall rules referencing them. Orphaned. We decommissioned them after a two-week observation period. This reduced complexity significantly.

Step 3: Rebuild in NSX 4. Rather than attempting a lift-and-shift of the NSX-V policy model, I rebuilt the relevant security groups natively in NSX 4 using tags on the new VCF workload VMs. The migration cutover for each VM included a step to apply the correct NSX tag before the network was activated — so the firewall policy was in place before the workload came online.

This approach — rebuild, don’t migrate — is slower than a scripted conversion, but it results in a clean policy model you actually understand. For a financial services firm heading into a SOC 2 audit, that tradeoff was obvious.

Setting Up Forklift on Alauda Container Platform

With the network foundation in place, the actual VM migration pipeline was built on Forklift Operator. For those unfamiliar: Forklift is a Kubernetes-native migration tool originally from the KubeVirt project that can migrate VMware VMs into KubeVirt-managed VMs (or in this case, into VKS-managed infrastructure) by intelligently copying disks using VDDK and converting them to PVCs.

The VDDK Setup

VDDK (VMware Virtual Disk Development Kit) is what Forklift uses to access VMware disk images efficiently. The setup is non-trivial because VDDK can’t be bundled in the Forklift image for licensing reasons — you have to provide it yourself as a container image.

The Containerfile:

FROM registry.access.redhat.com/ubi8/ubi-minimal
USER 1001
COPY vmware-vix-disklib-distrib /vmware-vix-disklib-distrib
RUN mkdir -p /opt
ENTRYPOINT ["cp", "-r", "/vmware-vix-disklib-distrib", "/opt"]

Build, push to the internal registry, and reference in the ForkliftController CR:

apiVersion: forklift.konveyor.io/v1beta1
kind: ForkliftController
metadata:
  name: forklift-controller
  namespace: konveyor-forklift
spec:
  vddk_job_image: registry.internal.client.com/infra/vddk-init:8.0.3

Note: VDDK version must match your vSphere version. We were migrating from vSphere 6.7 U3, which required VDDK 7.0.x for the source, but the target VCF 9 environment used VDDK 8.0.x. We ran a 7.0 VDDK image for the pilot phase and upgraded after confirming all source VMs had moved.

Registering the VMware Provider

apiVersion: forklift.konveyor.io/v1beta1
kind: Provider
metadata:
  name: vmware-prod-vc01
  namespace: konveyor-forklift
spec:
  type: vsphere
  url: https://vc01.infra.client.internal/sdk
  secret:
    name: vcenter-credentials
    namespace: konveyor-forklift

Credential secret contained the vCenter service account credentials. We created a read-only service account in vCenter with just enough permissions to enumerate inventory and access disks — no admin rights.

One gotcha: the Forklift provider sync can take 15–40 minutes the first time on a large inventory (we had ~4,200 managed objects in vCenter). I’d recommend letting it fully sync before building your first migration plan. Querying the Provider status:

kubectl get provider vmware-prod-vc01 -n konveyor-forklift -o jsonpath='{.status.conditions}'

Wait for Ready: True.

Network and Storage Mapping: Where Plans Meet Reality

Network Mapping

The source environment had 12 distributed port groups across two vDSwitches. The target NSX 4 environment had logical segments defined per application tier.

The mapping looked clean on paper. In practice:

• Three port groups had VMs from multiple application tiers mixed together. This is the legacy of “we had a spare port group” infrastructure decisions made in 2019.
• Two port groups used VLAN tagging that conflicted with how the new NSX logical segments were configured.

NetworkMap resource:

apiVersion: forklift.konveyor.io/v1beta1
kind: NetworkMap
metadata:
  name: prod-network-map
  namespace: konveyor-forklift
spec:
  map:
    - source:
        id: dvportgroup-1021
      destination:
        type: pod
    - source:
        id: dvportgroup-1047
      destination:
        type: multus
        name: prod-app-tier-01
        namespace: prod
  provider:
    source:
      name: vmware-prod-vc01
      namespace: konveyor-forklift
    destination:
      name: host
      namespace: konveyor-forklift

For the mixed-tier port groups, I worked with the network team to create intermediate segments and then used a two-phase approach: migrate into the intermediate segment, validate, then reconfigure the VM networking to the correct target segment. Extra work, but it kept the blast radius small.

Storage Mapping

The client had three storage classes on their vSAN cluster: Gold (all-flash, FTT=2), Silver (mixed, FTT=1), and Bronze (capacity tier). Target VCF 9 had vSAN ESA with its own storage policies.

apiVersion: forklift.konveyor.io/v1beta1
kind: StorageMap
metadata:
  name: prod-storage-map
  namespace: konveyor-forklift
spec:
  map:
    - source:
        id: datastore-101   # vSAN Gold
      destination:
        storageClass: vcf-vsan-esa-gold
    - source:
        id: datastore-102   # vSAN Silver
      destination:
        storageClass: vcf-vsan-esa-silver
    - source:
        id: datastore-201   # Legacy NFS (decommissioning)
      destination:
        storageClass: vcf-vsan-esa-silver
  provider:
    source:
      name: vmware-prod-vc01
      namespace: konveyor-forklift
    destination:
      name: host
      namespace: konveyor-forklift

Post-migration labeling note: After migration, PVCs created by Forklift needed to be labeled for the client’s internal chargeback and compliance tooling. This was a manual step we scripted:

for pvc in $(kubectl get pvc -n prod -o name | grep forklift); do
  kubectl label $pvc -n prod \
    migration.client.internal/source=vmware-prod-vc01 \
    migration.client.internal/wave=wave1 \
    compliance.client.internal/reviewed=false
done

The compliance.client.internal/reviewed=false label was important - it fed into a policy engine that prevented those VMs from entering the compliance boundary until a human reviewer signed off. Simple, but effective governance.

The Pilot Migration: When Theory Hits Metal

Week 5. Twenty non-production VMs. I chose them deliberately: two Windows Server 2019 instances with MSSQL, three Linux application servers, five utility VMs (DNS, NTP, monitoring agents), and a mix of disk sizes from 40GB to 2TB.

The 2TB disk was there specifically to stress-test the warm migration flow.

The Migration Plan

apiVersion: forklift.konveyor.io/v1beta1
kind: Plan
metadata:
  name: pilot-wave-00
  namespace: konveyor-forklift
spec:
  provider:
    source:
      name: vmware-prod-vc01
      namespace: konveyor-forklift
    destination:
      name: host
      namespace: konveyor-forklift
  map:
    network:
      name: prod-network-map
      namespace: konveyor-forklift
    storage:
      name: prod-storage-map
      namespace: konveyor-forklift
  vms:
    - id: vm-10412   # app-pilot-01
    - id: vm-10413   # app-pilot-02
    - id: vm-10417   # db-pilot-sql-01
  warm: true
  targetNamespace: staging

Setting warm: true means Forklift uses CBT (Change Block Tracking) to do incremental disk copies while the VM is still running. You get a precopy phase that runs for as long as you need, then a cutover window where the final delta is copied and the VM is started on the target.

What Broke

Problem 1: CBT not enabled on two VMs. Forklift requires Change Block Tracking to be enabled on the source VMs for warm migration. Two of our pilot VMs had CBT disabled — this had apparently been turned off as a troubleshooting step in 2022 and never re-enabled.

Fix: re-enable CBT via PowerCLI, then reset the CBT tracking file:

$vm = Get-VM -Name "app-pilot-01"
$spec = New-Object VMware.Vim.VirtualMachineConfigSpec
$spec.changeTrackingEnabled = $true
$vm.ExtensionData.ReconfigVM($spec)
# Snapshot and delete to reset CBT
$snap = New-Snapshot -VM $vm -Name "cbt-reset"
Remove-Snapshot -Snapshot $snap -Confirm:$false

Problem 2: The 2TB disk. The initial precopy took 11 hours. Fine — expected. But on the second incremental pass, the Forklift pod running the transfer was evicted because we had set resource limits too conservatively on the migration job pods. The transfer restarted from scratch.

Fix: update the ForkliftController to allow higher resource requests for VDDK transfer pods:

spec:
  vddk_job_requests_memory: "1Gi"
  vddk_job_limits_memory: "2Gi"
  vddk_job_requests_cpu: "500m"
  vddk_job_limits_cpu: "2"

Also moved to a dedicated node pool for migration workloads so the transfer pods wouldn’t compete with production scheduler pressure.

Problem 3: Windows Server VMware Tools version. The two Windows VMs had VMware Tools 11.0.6 — old enough that the Forklift conversion process flagged a compatibility warning. The VMs migrated successfully but came up with the wrong network driver loaded (vmxnet2 instead of vmxnet3). Spent two hours troubleshooting why one of the SQL instances was seeing packet loss before catching it.

Fix: upgrade VMware Tools to 12.x on all Windows VMs in-place before migration. Added this as a mandatory pre-flight check in the runbook.

The Pilot Results

All 20 VMs migrated successfully. The 2TB disk completed in a total precopy time of ~14 hours across two incremental passes. Cutover window for the full pilot batch: 41 minutes, including validation steps.

More importantly, I had a runbook with three pages of real gotchas documented. That runbook is what made Wave 1 go smoothly.

Wave 1: 130 VMs, 4 Weeknight Windows

With the pilot learnings incorporated, Wave 1 ran across four Tuesday/Wednesday nights over three weeks — low-traffic periods for a financial services firm where Monday mornings are high-stakes and Fridays have end-of-week settlement runs.

Pre-flight Checklist (The Real One)

Before any VM entered a Forklift plan, it passed this checklist:

• CBT enabled and verified (PowerCLI check)
• VMware Tools ≥ 12.0 installed
• No active snapshots (snapshots break CBT)
• VM not in a vSphere HA cluster with overrides that would conflict with DRS during precopy
• IP address documented in IPAM (even if static in-guest, we catalogued it)
• Application owner notified with cutover window and rollback criteria
• NSX 4 target segment pre-provisioned and tested (ping from existing VM on segment)
• Storage class capacity headroom confirmed (>20% free on target datastore)

The checklist sounds bureaucratic. It is. That’s the point. At scale, the thing that kills migrations isn’t technical failure — it’s the assumption that something is true when it isn’t.

Structuring the Migration Plans

Rather than one giant Plan resource with 130 VMs, I created plans with ~15–20 VMs each, grouped by application tier. This meant:

1. Failures were isolated — one plan’s issues didn’t block another’s
2. We could run two or three plans concurrently on different nights
3. Each plan had a clear owner from the application team

# Check plan status
kubectl get plan -n konveyor-forklift

NAME                    READY   EXECUTING   SUCCEEDED   FAILED   AGE
wave1-infra-services    True    False       True        False    12d
wave1-analytics-tier    True    False       True        False    9d
wave1-dev-tooling       True    True        False       False    2d
wave1-internal-apps     True    False       False       False    1d

The 2 AM Moment

Wave 1, Night 3. wave1-analytics-tier. 28 VMs. We were on the cutover step - final delta copy, then power-off source, power-on target.

VM analytics-ingest-07 came up on the target, but the application failed its healthcheck. The monitoring system paged the on-call app owner. I was watching the Forklift logs when the Slack message came in.

Root cause: the VM had a systemd service that started before the network interface was fully initialized (classic race condition), and the static IP we’d configured in NSX 4 took an extra 8 seconds to propagate to the logical segment. The service tried to bind to the IP on startup, failed, and didn’t retry.

Not a Forklift issue. Not a VMware issue. A 4-year-old application startup ordering bug that had never surfaced in the source environment because vmxnet3 on the old vSphere cluster initialized faster.

Fix in the moment: systemctl restart analytics-ingest.service. Service came up, healthcheck passed, we continued. Permanent fix: added a network-online.target dependency to the service unit file and documented it as a post-migration action for that application tier.

Wave 1 completed with 128 of 130 VMs successfully migrated. Two VMs were deferred: one had a custom SCSI controller configuration that required manual intervention, one was a legacy 32-bit Windows Server 2008 R2 instance that the application team couldn’t justify migrating (it got decommissioned instead, which was the right outcome).

VKS Enablement: Standing Up the Supervisor Cluster

Parallel to the VM migration waves, I was enabling VKS on the VCF 9 environment to prepare for the client’s net-new application workloads.

Enabling Workload Management

VKS enablement in VCF 9 starts with enabling Workload Management on a vSphere cluster. Prerequisites:

• vSphere cluster with DRS enabled
• NSX 4 overlay network (T1 gateway per namespace recommended)
• vSAN storage policy for supervisor etcd (separate from workload storage)
• Dedicated IP ranges for Supervisor Control Plane VMs (5 IPs for a production deployment)
• Load balancer integration (we used NSX Advanced Load Balancer / Avi)

The Supervisor Control Plane came up on three VMs (for HA) and took about 35 minutes to initialize. I’ve seen this take up to 90 minutes on first deployment depending on cluster size and storage latency — don’t panic if the vCenter “Enabling Workload Management” progress bar seems frozen.

Namespace Configuration

We created three vSphere Namespaces:

prod        - production workloads, TKC 1.28
staging     - pre-prod validation, TKC 1.27
platform    - shared services (monitoring, registry, GitOps tooling)

Each namespace had:

• Dedicated resource quotas (CPU/memory/storage)
• NSX T1 gateway
• Storage policy binding
• RBAC mapped to AD groups via the vCenter Identity Provider

Deploying the First Tanzu Kubernetes Cluster

apiVersion: cluster.x-k8s.io/v1beta1
kind: Cluster
metadata:
  name: prod-app-cluster-01
  namespace: prod
spec:
  clusterNetwork:
    pods:
      cidrBlocks: ["192.168.128.0/17"]
    services:
      cidrBlocks: ["10.96.0.0/12"]
  topology:
    class: tanzukubernetescluster
    version: v1.28.8---vmware.1-fips.1
    controlPlane:
      replicas: 3
    workers:
      machineDeployments:
        - class: node-pool
          name: worker-pool-01
          replicas: 6
          variables:
            overrides:
              - name: vmClass
                value: best-effort-2xlarge
              - name: storageClass
                value: vcf-vsan-esa-gold

First TKC provisioned in 18 minutes. I’ve since done this on a dozen engagements and that’s about average for a 3-control-plane + 6-worker deployment.

The Compliance Boundary: Phase 4

The final 50 VMs. Trading platform. Risk systems. Archival. The ones with names like compliance-vault-01 and trading-feed-primary.

Every change here required a CAB submission with 10 business days lead time. We submitted four change requests in parallel, staggered by one week, giving us a rolling four-week window for controlled migrations.

Each migration in this phase had:

1. Pre-migration parallel run period (5 business days): Target VM running alongside source, receiving replicated traffic via a read-only tap. Compliance team verified data integrity.
2. CAB-approved maintenance window: Saturday 10 PM — Sunday 6 AM
3. Rollback criteria: If target VM failed healthcheck within 30 minutes of cutover, automatic revert to source. This was scripted and tested.
4. Post-migration sign-off: Application owner + compliance officer jointly signed a checklist before the source VM was powered down permanently.

The longest migration in this phase was trading-feed-primary - a 6TB disk, warm migration precopy running for 4 days before the cutover window. Precopy delta at cutover time: 47GB. Cutover completed in 23 minutes.

All 50 compliance-boundary VMs migrated successfully. SOC 2 audit began six weeks after the final migration completed. We passed.

What I’d Do Differently

1. Automate the pre-flight checklist from day one. I built the PowerCLI/Ansible pre-flight automation in Week 3, after running the checks manually for the pilot. Should have been Week 1. Manual checklists at scale are a liability.

2. Build a migration status dashboard earlier. I wrote a simple script that queried Forklift Plan status and pushed it to a Grafana dashboard. Application owners loved it — they could see their VMs moving through precopy, incremental, and cutover phases without pinging me. Should have shipped this before Wave 1, not during it.

3. Plan for VMware Tools upgrades as a separate work stream. We touched ~80 VMs for VMware Tools upgrades before they could enter a migration plan. That’s non-trivial change management, especially in a regulated environment. In future engagements, I’ll request a VMware Tools audit and upgrade phase as a formal prerequisite with its own timeline and CAB approvals.

4. The NSX-V cleanup takes longer than you think. “We’ll clean up the old NSX-V objects after the migration” is a thing I said in Week 2. We were still decommissioning orphaned NSX-V rules in Week 15. Invest in the cleanup from the start.

Architecture Lessons That Don’t Fit in a Diagram

The technology in this engagement — VCF 9, Forklift, VKS, NSX 4 — is well-documented. The VMware and KubeVirt communities have good reference material. What isn’t documented is the organizational layer.

The most complex thing about migrating 200 VMs is not the VM migration. It’s the 200 application owners, the 14 compliance rules, the 3 change management processes, and the 1 executive who agreed to the project timeline without consulting the team that actually owns the Saturday maintenance windows.

The technical execution of this project was straightforward once the environment was understood. The architectural decisions that mattered most were organizational: how we structured the wave plan around business risk, how we gave application owners visibility without giving them enough access to accidentally break something, how we threaded the CAB process without letting it become a six-month bloat.

If you’re planning a migration like this, spend more time on the stakeholder model than on the YAML manifests. The manifests will work. The people will surprise you.

Devan McCormick is a Senior VMware Cloud Architect focused on enterprise infrastructure modernization, VCF deployments, and Kubernetes enablement. You can find him on GitHub at @devancormick.

If this was useful, consider following for more deep-dives on enterprise infrastructure work. And if you’re in the middle of something like this and it’s 2 AM — the analytics service restart will work. It always does.