Some scripts for Linux KVM and qcow2 management

Some scripts for Linux KVM and qcow2 management

When you have lots and lots of qcow2

One of my lab machines has over 100 qcow2 devices across 40 KVM virtual machines. And that is just one machine. FreeBSD, Fedora, Mint, Ubuntu, alpine, clones of clones.

Here is a quick perl script that actually lists the qcow2 images from the VM perspective. It shows devices and how much storage each VM is using.

vm_disk_report.pl

#!/usr/bin/perl
use strict;
use warnings;

# Execute virsh list --all command and capture the output
my @vms = `virsh list --all`;
shift @vms; # Remove the header line
shift @vms; # Remove the separator line

# Initialize a hash to store VM disk information
my %vm_disks;

# Iterate over each VM
foreach my $vm (@vms) {
    chomp $vm;
    # Use a regex to correctly parse the VM details
    if ($vm =~ /^\s*(-|\d+)\s+(\S+)\s+(running|paused|shutdown|shut off|crashed|pmsuspended)$/) {
        my ($id, $name, $state) = ($1, $2, $3);

        # Execute virsh domblklist command for each VM
        my @disks = `virsh domblklist $name`;
        shift @disks; # Remove the header line
        shift @disks; # Remove the separator line

        # Store disk information in the hash
        $vm_disks{$name} = \@disks;
    }
}

# Function to get actual disk size on disk in GB
sub get_disk_size {
    my ($disk) = @_;
    my $size = `stat --format=%b $disk 2>/dev/null`; # Get the number of 512-byte blocks allocated
    chomp $size;
    return $size ? sprintf("%.2f", $size * 512 / (1024 * 1024 * 1024)) : 'not found';
}

# Generate the report
print "VM Disk Report\n";
print "==============\n\n";

foreach my $vm (keys %vm_disks) {
    print "VM Name: $vm\n";
    print "-----------------\n";
    my $total_size = 0;
    foreach my $disk (@{$vm_disks{$vm}}) {
        chomp $disk;
        my ($target, $source) = split(/\s+/, $disk, 2);
        $target = defined $target ? $target : '';
        $source = defined $source ? $source : '';

        # Skip entries with no data
        next if $target eq '' && $source eq '';

        my $size_gb = get_disk_size($source);
        $total_size += $size_gb ne 'not found' ? $size_gb : 0;

        printf "Disk: %-4s source: %-80s size: %s GB\n", $target, $source, $size_gb;
    }
    print "\n";
    printf "Total size for %s: %.2f GB\n\n", $vm, $total_size;
}

The report looks like this:

VM Disk Report
==============

VM Name: Knoppixcore
-----------------
Disk:      source: sda      /home/jon2allen/Downloads/knoppix.qcow2                                 size: 13.58 GB

Total size for Knoppixcore: 13.58 GB

VM Name: jenkins
-----------------
Disk:      source: vda      /nvme1/jenkins.qcow2                                                    size: not found GB
Disk:      source: sda      -                                                                       size: not found GB

Total size for jenkins: 0.00 GB

VM Name: debian12
-----------------
Disk:      source: vda      /play/test_disk/debian12.qcow2                                          size: 3.03 GB
Disk:      source: sda      -                                                                       size: not found GB

Total size for debian12: 3.03 GB

VM Name: fedora37_server_with_zfs
-----------------
Disk:      source: vda      /home/jon2allen/.local/share/libvirt/images/fedora37_server_with_zfs.qcow2 size: 6.05 GB
Disk:      source: vdb      /play/test_disk/fedora_server_test1.qcow2                               size: 1.72 GB
Disk:      source: vdc      /play/test_disk/fedora_server_test2.qcow2                               size: 1.72 GB
Disk:      source: vdd      /play/test_disk/fedora_server_test3.qcow2                               size: 1.19 GB
Disk:      source: vde      /play/test_disk/fedora_server_test4.qcow2                               size: 1.19 GB
Disk:      source: vdf      /play/test_disk/fedora_server_test5.qcow2                               size: 1.19 GB
Disk:      source: vdg      /play/test_disk/fedorda_server_test7.qcow2                              size: 0.17 GB
Disk:      source: sda      -                                                                       size: not found GB

Total size for fedora37_server_with_zfs: 13.23 GB

As you can see I have qcow2 all over the place and an administrators nightmare.

When I was transferring a VM to a different machine. I discovered a wrinkle with this report…that I forgot about. Some qcow2 have a backing store qcow2 image.

What is a Backing store?

A qcow2 file with a backing store is typically used in snapshot or delta scenarios, where changes are stored in the overlay (the qcow2 file) while the original data remains in the backing file:

===================================================
                   READ OPERATION
===================================================

                 [ Guest OS Read Request ]
                        (Block X)
                            |
                            v
                  +-------------------+
                  |  Overlay (qcow2)  |
                  | Is Block X alloc- |
                  | ated here?        |
                  +-------------------+
                      /           \
                   [YES]         [NO]
                    /               \
                   v                 v
        +-----------------+   +-----------------+
        | Read Block X    |   | Fallback:       |
        | from the        |   | Read Block X    |
        | Overlay Image.  |   | from Backing.   |
        +-----------------+   +-----------------+

===================================================
                   WRITE OPERATION
===================================================

                [ Guest OS Write Request ]
                        (Block X)
                            |
                            v
                  +-------------------+
                  |  Overlay (qcow2)  |
                  | Is Block X alloc- |
                  | ated here yet?    |
                  +-------------------+
                      /           \
                   [YES]         [NO]
                    /               \
                   v                 v
        +-----------------+   +-----------------+
        | Overwrite data  |   | Allocate new    |
        | directly inside |   | space for X in  |
        | the Overlay.    |   | Overlay, write. |
        +-----------------+   +-----------------+
                                      |
              (The Backing Store is NEVER touched)

===================================================
             QCOW2 CLUSTER ARCHITECTURE
===================================================

       Guest OS sees a single seamless virtual disk
                       |   ^
                 Write |   | Read
                       v   |
          +------------------------------------+
          |           OVERLAY (qcow2)          |
          +------------------------------------+
          | Block 1 | Block 2 | Block 3 | ...  |
          | (New)   | (Empty) | (Modded)|      |
          +---|---------|---------|------------+
              |         |         |
   Write logic always   |         | Read logic pulls from
   forces data into     |         | overlay if data exists.
   the overlay.         |         |
                        |         |
              Read falls back     |
              if empty.           |
                        |         |
          +-------------|---------|------------+
          |             v         |            |
          |       BACKING STORE (Read-Only)    |
          +------------------------------------+
          | (Empty) | Block 2 | Block 3 | ...  |
          |         | (Orig)  | (Orig)  |      |
          +------------------------------------+

So, how do you know if a qcow2 has a backing store — use qemu-img info command and look for backing file.

image: freebsd12.3.backup
file format: qcow2
virtual size: 8 GiB (8589934592 bytes)
disk size: 13.2 GiB
cluster_size: 65536
backing file: /home/jon2allen/.local/share/libvirt/images/freebsd12.3.qcow2
backing file format: qcow2
Snapshot list:
ID        TAG               VM SIZE                DATE     VM CLOCK     ICOUNT
1         snapshot1-may-02      0 B 2024-05-02 22:26:49 00:06:10.599           
2         snapshot_6_04         0 B 2024-06-04 22:22:38 11:08:25.597           
Format specific information:
    compat: 1.1
    compression type: zlib
    lazy refcounts: false
    refcount bits: 16
    corrupt: false
    extended l2: false
Child node '/file':
    filename: freebsd12.3.backup
    protocol type: file
    file length: 13.2 GiB (14178648064 bytes)
    disk size: 13.2 GiB

Is there a way to know from a gui like virt-manager?

Well, it is not very consistent in the display. If the VM is running then it shows the backing store in the XML for the device.

screenshot of virt-manager running — can see backing store

However, if the VM is not running — you will not even be aware of its existance

device xml — VM not running.

So, I want to update the script to capture this backing store:

Key change is this

>     # Get backing file using qemu-img
>     my $info = `qemu-img info "$source" 2>/dev/null`;
>     my ($backing) = $info =~ /^backing file: (.*)$/m;

Also some other improvements — like using stat in perl rather than a system call.

vm_disk_report_enh.pl

#!/usr/bin/perl
use strict;
use warnings;

# Execute virsh list --all command and capture the output
my @vms = `virsh list --all`;
shift @vms; # Remove the header line
shift @vms; # Remove the separator line

# Initialize a hash to store VM disk information
my %vm_disks;

# Iterate over each VM
foreach my $vm (@vms) {
    chomp $vm;
    # Use a regex to correctly parse the VM details
    if ($vm =~ /^\s*(-|\d+)\s+(\S+)\s+(running|paused|shutdown|shut off|crashed|pmsuspended)$/) {
        my ($id, $name, $state) = ($1, $2, $3);

        # Execute virsh domblklist command for each VM
        my @disks = `virsh domblklist $name`;
        shift @disks; # Remove the header line
        shift @disks; # Remove the separator line

        # Store disk information in the hash
        $vm_disks{$name} = \@disks;
    }
}

# Function to get disk info using stat (size) and qemu-img (backing file)
sub get_disk_data {
    my ($source) = @_;
    return ('not found', 'none') unless $source && -e $source;

    # Get actual size on disk using stat (512-byte blocks allocated)
    my @stats = stat($source);
    my $blocks = $stats[12]; 
    my $size_gb = defined $blocks ? sprintf("%.2f", ($blocks * 512) / (1024**3)) : 'not found';

    # Get backing file using qemu-img
    my $info = `qemu-img info "$source" 2>/dev/null`;
    my ($backing) = $info =~ /^backing file: (.*)$/m;

    return ($size_gb, $backing || 'none');
}

# Generate the report
print "VM Disk Report\n";
print "==============\n\n";

foreach my $vm (sort keys %vm_disks) {
    print "VM Name: $vm\n";
    print "-----------------\n";
    my $total_size = 0;
    foreach my $disk (@{$vm_disks{$vm}}) {
        chomp $disk;
        my ($target, $source) = $disk =~ /^\s*(\S+)\s+(.*)$/;
        next unless $source;

        my ($size_gb, $backing) = get_disk_data($source);
        $total_size += ($size_gb ne 'not found') ? $size_gb : 0;

        printf "Disk: %-4s source: %-60s size: %7s GB\n", $target, $source, $size_gb;
        if ($backing ne 'none') {
            printf "      backing: %s\n", $backing;
        }
    }
    print "\n";
    printf "Total size for %s: %.2f GB\n\n", $vm, $total_size;
}

Now we have an accurate accounting of all qcow2 data.

VM Name: freebsd12.3
-----------------
Disk: vda  source: /play/test_disk/freebsd.qcow2                                size:   34.33 GB
Disk: vdb  source: /home/jon2allen/.local/share/libvirt/images/freebsd12.3.backup size:   13.20 GB
      backing: /home/jon2allen/.local/share/libvirt/images/freebsd12.3.qcow2
Disk: vdc  source: /play/test_disk/test500mb.qcow2                              size:    0.09 GB

Total size for freebsd12.3: 47.62 GB

if you move the backing store one must used the qemu-img rebase to update the reference.

proper syntax — qemu-img rebase -u -b <full path to new file> -F qcow2 <overlay.qcow2>

on2allen@jon-fedora:~/.local/share/libvirt/images$ qemu-img info freebsd12.3.backup
image: freebsd12.3.backup
file format: qcow2
virtual size: 8 GiB (8589934592 bytes)
disk size: 13.2 GiB
cluster_size: 65536
backing file: /home/jon2allen/.local/share/libvirt/images/freebsd12.3.qcow2
backing file format: qcow2
Snapshot list:
ID        TAG               VM SIZE                DATE     VM CLOCK     ICOUNT
1         snapshot1-may-02      0 B 2024-05-02 22:26:49 00:06:10.599           
2         snapshot_6_04         0 B 2024-06-04 22:22:38 11:08:25.597           
Format specific information:
    compat: 1.1
    compression type: zlib
    lazy refcounts: false
    refcount bits: 16
    corrupt: false
    extended l2: false
Child node '/file':
    filename: freebsd12.3.backup
    protocol type: file
    file length: 13.2 GiB (14178648064 bytes)
    disk size: 13.2 GiB
jon2allen@jon-fedora:~/.local/share/libvirt/images$ mv freebsd12.3.qcow2 freebsd12.3_backing.qcow2

...

jon2allen@jon-fedora:~/.local/share/libvirt/images$ qemu-img rebase -u -b /home/jon2allen/.local/share/libvirt/images/freebsd12.3_backing.qcow2  freebsd12.3.backup
qemu-img: Could not change the backing file to '/home/jon2allen/.local/share/libvirt/images/freebsd12.3_backing.qcow2': backing format must be specified
jon2allen@jon-fedora:~/.local/share/libvirt/images$ qemu-img rebase -u -b /home/jon2allen/.local/share/libvirt/images/freebsd12.3_backing.qcow2 -F qcow2  freebsd12.3.backup
jon2allen@jon-fedora:~/.local/share/libvirt/images$ qemu-img info freebsd12.3.backup
image: freebsd12.3.backup
file format: qcow2
virtual size: 8 GiB (8589934592 bytes)
disk size: 13.2 GiB
cluster_size: 65536
backing file: /home/jon2allen/.local/share/libvirt/images/freebsd12.3_backing.qcow2
backing file format: qcow2
Snapshot list:
ID        TAG               VM SIZE                DATE     VM CLOCK     ICOUNT
1         snapshot1-may-02      0 B 2024-05-02 22:26:49 00:06:10.599           
2         snapshot_6_04         0 B 2024-06-04 22:22:38 11:08:25.597           
Format specific information:
    compat: 1.1
    compression type: zlib
    lazy refcounts: false
    refcount bits: 16
    corrupt: false
    extended l2: false
Child node '/file':
    filename: freebsd12.3.backup
    protocol type: file
    file length: 13.2 GiB (14178648064 bytes)
    disk size: 13.2 GiB

And sure enough — virt-manager picks it up from the information metadata

after backing store rebase

domain_shutdown.sh

Shutdown all active domains. normal host Linux shutdown on many active VM can cause abnormal shutdowns, leading to fsck when booting back up if your VM is using older style filesystems. I mean that is what virtual machines were made for — running old stuff like it is 1999…

#!/bin/bash

logger "virsh shutting down VM"

for domain in $(virsh list --state-running --name); do
      virsh shutdown "$domain"
done

logger "virsh waiting 10 seconds"

sleep 14

logger "virsh shutdown complete"

The virsh shutdown is not synchronous. So, sleeping for a while is good thing. Either run before you poweroff or put it in your shutdown scripts..

Hopefully this information makes managing your qcow2 files a bit easier and makes it easier to move your disks from one machine to a more powerful machines with all that expensive RAM. And a bonus shutdown script.

AI generated — moving equipment to new racks in data cener.