Ceph OSD Replacement on OpenShift Data Foundation (ODF)

This article documents a Ceph OSD replacement procedure in an OpenShift Data Foundation (ODF) environment using Local Storage Operator (LSO) backed devices.

The steps cover:

• Safely stopping Ceph/ODF operators
• Forcibly removing a failed OSD
• Cleaning the underlying disk at the node level
• Releasing and re-creating the Local PV
• Allowing ODF to automatically re-provision the OSD
• Monitoring Ceph recovery until the cluster returns to HEALTH_OK

This procedure is typically used when:

• An OSD is failed or stuck
• A disk needs to be replaced or reinitialized
• OSD prepare jobs are failing

In this case there is “osd-2” got error after node revision.

Step 1: Scale Down ODF and Ceph Operators

To prevent reconciliation conflicts during OSD removal, scale down the operators first.

oc project openshift-storage

oc scale deployment ocs-operator rook-ceph-operator --replicas=0

Scale down the affected OSD deployment:

oc scale -n openshift-storage deployment rook-ceph-osd-2 --replicas=0

Verify remaining OSDs:

oc get pod | grep osd

[root@bastionocp ~]# oc get pod | grep osd
rook-ceph-osd-0-7cc7bd4b4f-2vqft                                  2/2     Running   0               5d20h
rook-ceph-osd-1-876cbcfb5-vdbgn                                   2/2     Running   0               5d20h

Step 2: Clean Up Failed OSD Prepare Jobs

Failed or stale rook-ceph-osd-prepare jobs can block re-provisioning.

oc get jobs -n openshift-storage
oc delete jobs --all -n openshift-storage

[root@bastionocp ~]# oc get jobs -n openshift-storage
NAME                                                     STATUS     COMPLETIONS   DURATION   AGE
rook-ceph-osd-prepare-20624464a1bad18aadb404817532599b   Failed     0/1           5d19h      5d19h
rook-ceph-osd-prepare-9da48916675e56333a3502e8e1bccf9e   Complete   1/1           15s        145d
rook-ceph-osd-prepare-ba68212104e31ed4c15b7b638d669835   Complete   1/1           18s        145d
rook-ceph-osd-prepare-df2c8315759abe198f540c2c9931b076   Complete   1/1           19s        145d
[root@bastionocp ~]# oc delete jobs --all -n openshift-storage
job.batch "rook-ceph-osd-prepare-20624464a1bad18aadb404817532599b" deleted
job.batch "rook-ceph-osd-prepare-9da48916675e56333a3502e8e1bccf9e" deleted
job.batch "rook-ceph-osd-prepare-ba68212104e31ed4c15b7b638d669835" deleted
job.batch "rook-ceph-osd-prepare-df2c8315759abe198f540c2c9931b076" deleted

Step 3: Remove the Failed OSD from Ceph

Use the built-in OSD removal job template.

oc process -n openshift-storage ocs-osd-removal \
  -p FAILED_OSD_IDS=2 \
  -p FORCE_OSD_REMOVAL=true | \
  oc create -n openshift-storage -f -

Monitor the job:

oc get job -n openshift-storage

[root@bastionocp ~]# oc get job
NAME                  STATUS    COMPLETIONS   DURATION   AGE
ocs-osd-removal-job   Running   0/1           7s         7s
[root@bastionocp ~]# oc get job
NAME                  STATUS     COMPLETIONS   DURATION   AGE
ocs-osd-removal-job   Complete   1/1           10s        20s

Confirm successful removal:

oc logs -l job-name=ocs-osd-removal-job \
  -n openshift-storage --tail=-1 | \
  egrep -i 'completed removal'

Expected output:

[root@bastionocp ~]# oc logs -l job-name=ocs-osd-removal-job -n openshift-storage --tail=-1 | egrep -i 'completed removal'
2026-01-19 04:47:27.970456 I | cephosd: completed removal of OSD 2

Step 4: Clean the Disk on the Node

Log into the affected node to wipe the disk completely.

oc debug node/odf-2.dev.ocp.bankabc.co.id
chroot /host

Identify the disk:

lsblk

sh-5.1# lsblk
NAME   MAJ:MIN RM   SIZE RO TYPE MOUNTPOINTS
sda      8:0    0   500G  0 disk
sdb      8:16   0   200G  0 disk
|-sdb1   8:17   0     1M  0 part
|-sdb2   8:18   0   127M  0 part
|-sdb3   8:19   0   384M  0 part /boot
`-sdb4   8:20   0 199.5G  0 part /var
                                 /sysroot/ostree/deploy/rhcos/var
                                 /sysroot
                                 /usr
                                 /etc

Clean the disk (example uses /dev/sda):

⚠️ Warning: Ensure this is the correct OSD disk before wiping.

sgdisk -Z /dev/sda
wipefs -a /dev/sda

Step 5: Remove Old PVC and Release the Local PV

Identify the PVC tied to the removed OSD:

Before (Make sure you have a local pv in “Available” state)

[root@bastionocp ~]# oc get pv | grep local
local-pv-32b6af3f                          500Gi      RWO            Delete           Bound    openshift-storage/ocs-deviceset-lso-volumeset-0-data-0p6bqd        lso-volumeset                 <unset>                          5d19h
local-pv-8cd6bd8a                          500Gi      RWO            Delete           Bound    openshift-storage/ocs-deviceset-lso-volumeset-0-data-16crf7        lso-volumeset                 <unset>                          146d
local-pv-ea50264f                          500Gi      RWO            Delete           Bound    openshift-storage/ocs-deviceset-lso-volumeset-0-data-2p82pb        lso-volumeset                 <unset>                          146d

[root@bastionocp ~]# oc get pvc -n openshift-storage | grep deviceset
ocs-deviceset-lso-volumeset-0-data-0p6bqd   Bound    local-pv-32b6af3f                          500Gi      RWO            lso-volumeset                 <unset>                 5d19h
ocs-deviceset-lso-volumeset-0-data-16crf7   Bound    local-pv-8cd6bd8a                          500Gi      RWO            lso-volumeset                 <unset>                 145d
ocs-deviceset-lso-volumeset-0-data-2p82pb   Bound    local-pv-ea50264f                          500Gi      RWO            lso-volumeset                 <unset>                 145d

oc get pvc -n openshift-storage | grep deviceset

Delete the PVC:

oc delete pvc ocs-deviceset-lso-volumeset-0-data-0p6bqd -n openshift-storage

Verify PV status changes to Released:

oc get pv | grep local

After (local-pv-32b6af3f is in released status)

[root@bastionocp ~]# oc get pv | grep local
local-pv-32b6af3f                          500Gi      RWO            Delete           Released   openshift-storage/ocs-deviceset-lso-volumeset-0-data-0p6bqd        lso-volumeset                 <unset>                          5d19h
local-pv-8cd6bd8a                          500Gi      RWO            Delete           Bound      openshift-storage/ocs-deviceset-lso-volumeset-0-data-16crf7        lso-volumeset                 <unset>                          146d
local-pv-ea50264f                          500Gi      RWO            Delete           Bound      openshift-storage/ocs-deviceset-lso-volumeset-0-data-2p82pb        lso-volumeset                 <unset>                          146d

Step 6: Allow ODF to Re-Provision the OSD

oc scale deployment ocs-operator rook-ceph-operator --replicas=1

Once:

• The disk is clean
• PVC is deleted
• LocalVolumeSet still exists

ODF will automatically:

• Recreate the local PV
• Run a new rook-ceph-osd-prepare job
• Deploy a new OSD pod

Verify:

oc get pod -n openshift-storage | egrep "mon|osd"

[root@bastionocp ~]# oc get pod -n openshift-storage | egrep "mon|osd"
ocs-osd-removal-job-s9m9x                                         0/1     Completed   0               17m
rook-ceph-mon-b-794f7ddf88-mt4j8                                  2/2     Running     0               5d21h
rook-ceph-mon-c-867bc96d5b-hbn2q                                  2/2     Running     0               5d19h
rook-ceph-mon-d-767c77df6-ggpth                                   2/2     Running     0               5d21h
rook-ceph-osd-0-7cc7bd4b4f-2vqft                                  2/2     Running     0               5d21h
rook-ceph-osd-1-876cbcfb5-vdbgn                                   2/2     Running     0               5d21h
rook-ceph-osd-2-6979fd6d56-9s977                                  2/2     Running     0               77s
rook-ceph-osd-prepare-1c1005a579cc58ec50ac2b29ce80f633-hw7lk      0/1     Completed   0               89s

Step 7: Monitor Ceph Health and Recovery

Check Ceph status:

oc -n openshift-storage exec -it deploy/rook-ceph-tools -- ceph status

During recovery, you may see:

• HEALTH_WARN
• Degraded or undersized PGs
• Backfill and recovery activity

[root@bastionocp ~]# oc -n openshift-storage exec -it deploy/rook-ceph-tools -- ceph status
  cluster:
    id:     699801d6-12c1-4411-8174-ceebe788fbc8
    health: HEALTH_WARN
            Degraded data redundancy: 6267/29079 objects degraded (21.552%), 28 pgs degraded, 28 pgs undersized
            25 pgs not deep-scrubbed in time

  services:
    mon: 3 daemons, quorum b,c,d (age 5d)
    mgr: a(active, since 96s), standbys: b
    mds: 1/1 daemons up, 1 hot standby
    osd: 3 osds: 3 up (since 92s), 3 in (since 108s); 28 remapped pgs
    rgw: 1 daemon active (1 hosts, 1 zones)

  data:
    volumes: 1/1 healthy
    pools:   12 pools, 185 pgs
    objects: 9.69k objects, 29 GiB
    usage:   58 GiB used, 1.4 TiB / 1.5 TiB avail
    pgs:     0.541% pgs not active
             6267/29079 objects degraded (21.552%)
             156 active+clean
             28  active+undersized+degraded+remapped+backfill_wait
             1   peering

  io:
    client:   1.5 KiB/s rd, 308 KiB/s wr, 2 op/s rd, 15 op/s wr
    recovery: 99 MiB/s, 0 keys/s, 38 objects/s

This is expected behavior.

Just wait …

Step 8: Verify Final Cluster Health

After recovery completes, check again:

oc rsh -n openshift-storage \
$(oc get pods -n openshift-storage -o name -l app=rook-ceph-operator)

export CEPH_ARGS='-c /var/lib/rook/openshift-storage/openshift-storage.config'
ceph -s

sh-5.1$ ceph -s
  cluster:
    id:     699801d6-12c1-4411-8174-ceebe788fbc8
    health: HEALTH_OK

  services:
    mon: 3 daemons, quorum b,c,d (age 5d)
    mgr: a(active, since 28m), standbys: b
    mds: 1/1 daemons up, 1 hot standby
    osd: 3 osds: 3 up (since 28m), 3 in (since 28m)
    rgw: 1 daemon active (1 hosts, 1 zones)

  data:
    volumes: 1/1 healthy
    pools:   12 pools, 185 pgs
    objects: 9.69k objects, 29 GiB
    usage:   73 GiB used, 1.4 TiB / 1.5 TiB avail
    pgs:     185 active+clean

  io:
    client:   1.4 KiB/s rd, 247 KiB/s wr, 2 op/s rd, 16 op/s wr