ZFS operations: replacing a mirrored disk

You have a zpool with a mirror vdev. For example:

root@server:~#zpool status
  pool: tank
 state: ONLINE
  scan: scrub repaired 0B in 02:09:18 with 0 errors on Sun May 10 02:33:19 2026
config:

        NAME                                      STATE     READ WRITE CKSUM
        tank                                      ONLINE       0     0     0
          mirror-0                                ONLINE       0     0     0
            scsi-0QEMU_QEMU_HARDDISK_drive-scsi1  ONLINE       0     0     0
            scsi-SQEMU_QEMU_HARDDISK_WD-AS04M8BR  ONLINE       0     0     0

errors: No known data errors

Your block device config looks something like this:

root@server:~# lsblk -i
NAME                      MAJ:MIN RM   SIZE RO TYPE MOUNTPOINTS
sda                         8:0    0    32G  0 disk
|-sda1                      8:1    0     1M  0 part
|-sda2                      8:2    0     2G  0 part /boot
`-sda3                      8:3    0    30G  0 part
  `-ubuntu--vg-ubuntu--lv 252:0    0    30G  0 lvm  /
sdb                         8:16   0 931.5G  0 disk
|-sdb1                      8:17   0 931.5G  0 part
`-sdb9                      8:25   0     8M  0 part
sdc                         8:32   0   3.6T  0 disk
|-sdc1                      8:33   0   3.6T  0 part
`-sdc9                      8:41   0     8M  0 part

So yes, you effectively have a mirror vdev with two disks of different size. You are now intent on swapping out the 1TB disk with another 4TB disk to grow the pool.

A good strategy is to add the new disks to the mirror vdev making it a 3-way mirror. This ensures resilvering completes without reliance on a single disk.

ZFS (on Ubuntu 24.04, running as a VM on Proxmox VE and with physical disks passed-through) will accept a new unpartitioned drive (sdd in this case) and will create a partition and add the disk to the mirror vdev.

Here’s what it looks like:

root@server:~# ls -al /dev/disk/by-id/
total 0
drwxr-xr-x  2 root root 420 Jun  2 20:10 .
drwxr-xr-x 10 root root 200 Jun  2 20:10 ..
lrwxrwxrwx  1 root root   9 Jun  2 20:10 ata-QEMU_DVD-ROM_QM00003 -> ../../sr0
lrwxrwxrwx  1 root root  10 Jun  2 20:10 dm-name-ubuntu--vg-ubuntu--lv -> ../../dm-0
lrwxrwxrwx  1 root root  10 Jun  2 20:10 dm-uuid-LVM-15Agpmt1sBiG3Xh1pbnwmJPOsOPHJiabfDa1fAfh0ZDpohpDNUFGhpatKkjkmbv3 -> ../../dm-0
lrwxrwxrwx  1 root root  10 Jun  2 20:10 lvm-pv-uuid-ECLQuF-8knE-sCm8-8Yd7-yRR3-GOAd-GLJOBn -> ../../sda3
lrwxrwxrwx  1 root root   9 Jun  2 20:10 scsi-0QEMU_QEMU_HARDDISK_drive-scsi0 -> ../../sda
lrwxrwxrwx  1 root root  10 Jun  2 20:10 scsi-0QEMU_QEMU_HARDDISK_drive-scsi0-part1 -> ../../sda1
lrwxrwxrwx  1 root root  10 Jun  2 20:10 scsi-0QEMU_QEMU_HARDDISK_drive-scsi0-part2 -> ../../sda2
lrwxrwxrwx  1 root root  10 Jun  2 20:10 scsi-0QEMU_QEMU_HARDDISK_drive-scsi0-part3 -> ../../sda3
lrwxrwxrwx  1 root root   9 Jun  2 20:10 scsi-0QEMU_QEMU_HARDDISK_drive-scsi1 -> ../../sdb
lrwxrwxrwx  1 root root  10 Jun  2 20:10 scsi-0QEMU_QEMU_HARDDISK_drive-scsi1-part1 -> ../../sdb1
lrwxrwxrwx  1 root root  10 Jun  2 20:10 scsi-0QEMU_QEMU_HARDDISK_drive-scsi1-part9 -> ../../sdb9
lrwxrwxrwx  1 root root   9 Jun  2 20:10 scsi-0QEMU_QEMU_HARDDISK_drive-scsi2 -> ../../sdc
lrwxrwxrwx  1 root root  10 Jun  2 20:10 scsi-0QEMU_QEMU_HARDDISK_drive-scsi2-part1 -> ../../sdc1
lrwxrwxrwx  1 root root  10 Jun  2 20:10 scsi-0QEMU_QEMU_HARDDISK_drive-scsi2-part9 -> ../../sdc9
lrwxrwxrwx  1 root root   9 Jun  2 20:10 scsi-0QEMU_QEMU_HARDDISK_drive-scsi3 -> ../../sdd
lrwxrwxrwx  1 root root   9 Jun  2 20:10 scsi-SQEMU_QEMU_HARDDISK_WD-AS04M8BR -> ../../sdc
lrwxrwxrwx  1 root root  10 Jun  2 20:10 scsi-SQEMU_QEMU_HARDDISK_WD-AS04M8BR-part1 -> ../../sdc1
lrwxrwxrwx  1 root root  10 Jun  2 20:10 scsi-SQEMU_QEMU_HARDDISK_WD-AS04M8BR-part9 -> ../../sdc9
lrwxrwxrwx  1 root root   9 Jun  2 20:10 scsi-SQEMU_QEMU_HARDDISK_WD-BS08GVBH -> ../../sdd

root@server:~# lsblk -i
NAME                      MAJ:MIN RM   SIZE RO TYPE MOUNTPOINTS
sda                         8:0    0    32G  0 disk
|-sda1                      8:1    0     1M  0 part
|-sda2                      8:2    0     2G  0 part /boot
`-sda3                      8:3    0    30G  0 part
  `-ubuntu--vg-ubuntu--lv 252:0    0    30G  0 lvm  /
sdb                         8:16   0 931.5G  0 disk
|-sdb1                      8:17   0 931.5G  0 part
`-sdb9                      8:25   0     8M  0 part
sdc                         8:32   0   3.6T  0 disk
|-sdc1                      8:33   0   3.6T  0 part
`-sdc9                      8:41   0     8M  0 part
sdd                         8:48   0   3.6T  0 disk
sr0                        11:0    1  1024M  0 rom

Whilst the zpool mirror vdev is currently using -part1 partitions in the mirror, zfs automatically creates partitioning on sdd when adding it to the mirror through the ‘zfs attach’ command.

So before attaching let’s look at the zpool status one more time:

root@server:~# zpool status
  pool: tank
 state: ONLINE
  scan: scrub repaired 0B in 02:09:18 with 0 errors on Sun May 10 02:33:19 2026
config:

        NAME                                      STATE     READ WRITE CKSUM
        tank                                      ONLINE       0     0     0
          mirror-0                                ONLINE       0     0     0
            scsi-0QEMU_QEMU_HARDDISK_drive-scsi1  ONLINE       0     0     0
            scsi-SQEMU_QEMU_HARDDISK_WD-AS04M8BR  ONLINE       0     0     0

errors: No known data errors

root@server:~# zpool status -P tank
  pool: tank
 state: ONLINE
  scan: scrub repaired 0B in 02:09:18 with 0 errors on Sun May 10 02:33:19 2026
config:

        NAME                                                            STATE     READ WRITE CKSUM
        tank                                                            ONLINE       0     0     0
          mirror-0                                                      ONLINE       0     0     0
            /dev/disk/by-id/scsi-0QEMU_QEMU_HARDDISK_drive-scsi1-part1  ONLINE       0     0     0
            /dev/disk/by-id/scsi-SQEMU_QEMU_HARDDISK_WD-AS04M8BR-part1  ONLINE       0     0     0

errors: No known data errors

So, the theory is that by referring to one mirror-0 member and attaching to it, zfs will take the raw disk, partition it, select the relevant partition and attach that to the mirror (making it a 3-way mirror).

root@server:~# zpool attach tank scsi-SQEMU_QEMU_HARDDISK_WD-AS04M8BR /dev/disk/by-id/scsi-SQEMU_QEMU_HARDDISK_WD-BS08GVBH

root@server:~# zpool status -P tank
  pool: tank
 state: ONLINE
status: One or more devices is currently being resilvered.  The pool will
        continue to function, possibly in a degraded state.
action: Wait for the resilver to complete.
  scan: resilver in progress since Tue Jun  2 20:46:08 2026
        46.6G / 852G scanned at 1.19G/s, 0B / 852G issued
        0B resilvered, 0.00% done, no estimated completion time
config:

        NAME                                                            STATE     READ WRITE CKSUM
        tank                                                            ONLINE       0     0     0
          mirror-0                                                      ONLINE       0     0     0
            /dev/disk/by-id/scsi-0QEMU_QEMU_HARDDISK_drive-scsi1-part1  ONLINE       0     0     0
            /dev/disk/by-id/scsi-SQEMU_QEMU_HARDDISK_WD-AS04M8BR-part1  ONLINE       0     0     0
            /dev/disk/by-id/scsi-SQEMU_QEMU_HARDDISK_WD-BS08GVBH-part1  ONLINE       0     0     0

errors: No known data errors
root@immichserver:~# lsblk -i
NAME                      MAJ:MIN RM   SIZE RO TYPE MOUNTPOINTS
sda                         8:0    0    32G  0 disk
|-sda1                      8:1    0     1M  0 part
|-sda2                      8:2    0     2G  0 part /boot
`-sda3                      8:3    0    30G  0 part
  `-ubuntu--vg-ubuntu--lv 252:0    0    30G  0 lvm  /
sdb                         8:16   0 931.5G  0 disk
|-sdb1                      8:17   0 931.5G  0 part
`-sdb9                      8:25   0     8M  0 part
sdc                         8:32   0   3.6T  0 disk
|-sdc1                      8:33   0   3.6T  0 part
`-sdc9                      8:41   0     8M  0 part
sdd                         8:48   0   3.6T  0 disk
|-sdd1                      8:49   0   3.6T  0 part
`-sdd9                      8:57   0     8M  0 part
sr0                        11:0    1  1024M  0 rom

So this works as hypothesized, but I don’t have a good reference to documentation that explains this behaviour. This is something that has been observed in a lab setup and appears to hold true over several operations.

Once the resilvering completes, I use ‘zpool detach’ to remove the 931.5G (1TB) partition from the mirror. This allows me to, in turn, power off the server and physically remove the unused disk and completes the replacement.