Revamping a Three-Node OpenShift Cluster: A Smooth Ride (Mostly!)

Hey there tech adventurers and OpenShift enthusiasts! Ever wondered how a professional cloud team revamps an OpenShift cluster architecture without losing their minds — or their data? Grab your favorite beverage and join us for a journey through a real-world scenario we recently tackled: redesigning a three-node OpenShift cluster. Let’s dive right in, but don’t worry — we’ll keep things light, engaging, and yes, even fun!

The Challenge: Why Even Bother?

Imagine you’re cruising smoothly with your OpenShift cluster, but suddenly realize it’s time to grow up a bit. Maybe you need better scalability, improved data services, or simply want to avoid that dreaded “it worked on my machine!” scenario. Whatever the reason, you’ve hit the moment every IT pro secretly dreads: architecture redesign.

The Game Plan

Redesigning an OpenShift architecture isn’t as straightforward as swapping parts in your gaming PC (though we wish it was!). To keep chaos at bay, we broke the task down into bite-sized chunks:

• Preparation & Verification (aka, the “Safety Nets”)
• Node Management Magic
• Networking & Storage tweaks (ODF and ETCD)
• Master Node Makeovers

Sound complicated? Trust us, it’s simpler (and more fun) than it sounds!

Preparation: Because Hope Isn’t a Strategy

You wouldn’t skydive without a parachute, right? Similarly, diving into cluster redesign without backups and safety nets is a bad idea. Our mantra: prepare, verify, and then implement.

• Proactive Case: First things first, open a proactive support case. Always better safe than sorry.
• Backup, Backup, Backup! ETCD and data backups are crucial. Think of it as creating save-points in a game before tackling a boss fight.
• Disable resource-heavy workloads temporarily — this means you, Kyverno!
• Run tests, confirm cluster health, and verify stability — basically, ensure everything is ready to rock.

The Current Configuration and the Goal

Current Architecture

Final Destination

The Main Act: Cluster Redesign in Action

Step 1: Remove ODF Member (Goodbye Old Node!)

● Define the node to be replaced

export NODE=<node_name>

● Enable the tools pod for ODF

oc patch OCSInitialization ocsinit -n openshift-storage --type json --patch '[{
  "op": "replace",
  "path": "/spec/enableCephTools",
  "value": true
}]'

● Check the Ceph cluster health using oc rsh

oc rsh -n openshift-storage $(oc get pods -n openshift-storage | grep tools | awk '{print $1}') ceph -s

● Identify the node and view its labels

oc get node $NODE -o json | grep -A 2 '"cluster.ocs.openshift.io/openshift-storage"'

● Identify OSD pods on the node

oc get node $NODE -o json | grep -A 2 '"cluster.ocs.openshift.io/openshift-storage"'

● Identify the OSD pods running on the node you need to replace

oc get pods -n openshift-storage -o wide | grep -i $NODE

● Scale down the deployments for the affected pods (replace <X> with your deployment identifier)

oc scale deployment rook-ceph-mon-<X> --replicas=0 -n openshift-storage
oc scale deployment rook-ceph-osd-<X> --replicas=0 -n openshift-storage
oc scale deployment --selector=app=rook-ceph-crashcollector,node=$NODE
--replicas=0 -n openshift-storage

● Mark the node as unschedulable

oc adm cordon $NODE

● Drain the node

oc adm drain $NODE --force --delete-emptydir-data=true --ignore-daemonsets

● Shutdown the node

Step 2: Removing ETCD Member (The Cleanup Crew!)

● Define the node to be replaced

oc get nodes -l node-role.kubernetes.io/master | grep "NotReady"

● Choose a pod that is not on the affected node

oc -n openshift-etcd get pods -l k8s-app=etcd

● Run the following command

oc rsh -n openshift-etcd <pod name>

● View the member list

etcdctl member list -w table
etcdctl endpoint status -w table

● Remove the unhealthy etcd member

etcdctl member remove <member_id>

● View the member list again and verify that the member was removed

etcdctl member list -w table

● Turn off the quorum guard

oc patch etcd/cluster --type=merge -p '{
  "spec": {"unsupportedConfigOverrides": {"useUnsupportedUnsafeNonHANonProductionUnstableEtcd": true}}
}'

● Delete the affected node

echo $NODE
oc delete node $NODE

● Remove the old secrets for the unhealthy etcd member — List the

secrets for the unhealthy etcd member that was removed

oc get secrets -n openshift-etcd | grep $NODE

● Delete the secrets for the unhealthy etcd member

oc delete secret -n openshift-etcd etcd-peer-$NODE
oc delete secret -n openshift-etcd etcd-serving-$NODE
oc delete secret -n openshift-etcd etcd-serving-metrics-$NODE

Step 3: Re-create Control Plane Node (Bringing in Reinforcements!)

● Create and download a new ISO from the Assisted installer service

● Boot the new VM node from the ISO (the VM will be on the same Vlan)

NOTE: In the next steps, we will create a BM CR. This is necessary for the

new Master node (even in our case where the Master will be a VM).

● Once the node gets the IP address, the new agent is created in the

Hub cluster.

oc get agent.agent-install.openshift.io -n <namespace -> cluster name > -o json | jq -r
'
.items[] | "\(.metadata.name) \(.spec.hostname)"'

● Please note that you need to annotate the agent

oc edit agent <> -n <>

spec:
  approved: true
  clusterDeploymentName:
    name: <>
    namespace: <>
  role: master

● The installation process is now in the UI -> tab ‘All clusters -> cluster

● Confirm the master node is ready

oc get node

You can also see that etcd pods on the newly added master are

crashlooping.

This is expected behavior.

● Add BareMetalHost object

apiVersion: metal3.io/v1alpha1
kind: BareMetalHost
metadata:
  name: <NEW_MASTER>
  namespace: openshift-machine-api
  annotations: {}

spec:
  automatedCleaningMode: metadata
  bootMACAddress: 00:00:00:00:00:00
  bootMode: UEFI
  customDeploy:
    method: install_coreos
  externallyProvisioned: true
  online: true
  userData:
    name: master-user-data-managed
    namespace: openshift-machine-api

● Add Machine object

In order to get the value of <cluster_name-xxxxx> run the command

oc get infrastructures cluster -o jsonpath='{.status.infrastructureName}'

apiVersion: machine.openshift.io/v1beta1
kind: Machine
metadata:
  annotations:
    machine.openshift.io/instance-state: externally provisioned
    metal3.io/BareMetalHost: openshift-machine-api/<NEW_MASTER>
  finalizers:
    - machine.machine.openshift.io
  generation: 3
  labels:
    machine.openshift.io/cluster-api-cluster: <cluster-name-xxxxx>
    machine.openshift.io/cluster-api-machine-role: master
    machine.openshift.io/cluster-api-machine-type: master
  name: <NEW_MASTER>
  namespace: openshift-machine-api

spec:
  metadata: {}
  providerSpec:
    value:
      apiVersion: baremetal.cluster.k8s.io/v1alpha1
      customDeploy:
        method: install_coreos
      hostSelector: {}
      image:
        checksum: ""
        url: ""
      kind: BareMetalMachineProviderSpec
      userData:
        name: master-user-data-managed

● Link BMH and Machine and Node using the magic script

We need the Machine object to have the IP of the Node in its .status

field. This is because of how the etcd-operator accepts members of

the cluster. In order to achieve that, a link between BareMetalHost,

Machine, and Node is needed. The script below takes care of it:

#!/bin/bash
# Credit goes to https://bugzilla.redhat.com/show_bug.cgi?id=1801238.
# This script will link a Machine object and a Node object.
# This is needed in order to have the IP address of the Node present in the status of the Machine.

set -x
set -e

machine="$1"
node="$2"

if [ -z "$machine" ] || [ -z "$node" ]; then
    echo "Usage: $0 MACHINE NODE"
    exit 1
fi

uid=$(echo "$node" | cut -f1 -d':')
node_name=$(echo "$node" | cut -f2 -d':')

oc proxy &
proxy_pid=$!

function kill_proxy {
    kill "$proxy_pid"
}
trap kill_proxy EXIT SIGINT

HOST_PROXY_API_PATH="http://localhost:8001/apis/metal3.io/v1alpha1/namespaces/openshift-machine-api/baremetalhosts"

function wait_for_json() {
    local name="$1"
    local url="$2"
    local timeout="$3"
    shift 3
    local curl_opts=("$@")

    echo -n "Waiting for $name to respond"
    local start_time
    start_time=$(date +%s)

    until curl -g -X GET "$url" "${curl_opts[@]}" 2>/dev/null | jq . >/dev/null; do
        echo -n "."
        local curr_time
        curr_time=$(date +%s)
        local time_diff=$((curr_time - start_time))
        if [[ $time_diff -gt $timeout ]]; then
            echo -e "\nTimed out waiting for $name"
            return 1
        fi
        sleep 5
    done
    echo " Success!"
    return 0
}

wait_for_json oc_proxy "${HOST_PROXY_API_PATH}" 10 -H "Content-Type: application/json" -H "Accept: application/json"

addresses=$(oc get node -n openshift-machine-api "${node_name}" -o json | jq -c '.status.addresses')
machine_data=$(oc get machine -n openshift-machine-api -o json "${machine}")

host=$(echo "$machine_data" | jq '.metadata.annotations["metal3.io/BareMetalHost"]' | cut -f2 -d/ | sed 's/"/ /g')

if [ -z "$host" ]; then
    echo "Machine $machine is not linked to a host yet." 1>&2
    exit 1
fi

# The address structure on the host doesn't match the node,
# so extract the values we need into separate variables.
hostname=$(echo "${addresses}" | jq '.[] | select(.type == "Hostname") | .address' | sed 's/"/ /g')
ipaddr=$(echo "${addresses}" | jq '.[] | select(.type == "InternalIP") | .address' | sed 's/"/ /g')

host_patch='{
  "status": {
    "hardware": {
      "hostname": "'"${hostname}"'",
      "nics": [
        {
          "name": "eth1",
          "ip": "'"${ipaddr}"'",
          "mac": "00:00:00:00:00:00",
          "model": "unknown",
          "speedGbps": 10,
          "vlanId": 0,
          "pxe": true
        }
      ],
      "systemVendor": {
        "manufacturer": "Red Hat",
        "productName": "",
        "serialNumber": ""
      },
      "firmware": {
        "bios": {
          "date": "04/01/2014",
          "vendor": "SeaBIOS",
          "version": "1.11.0-2.el7"
        }
      },
      "ramMebibytes": 0,
      "storage": [],
      "cpu": {
        "arch": "x86_64",
        "model": "Intel(R) Xeon(R) CPU E5-2630 v4 @ 2.20GHz",
        "clockMegahertz": 2199.998,
        "count": 4,
        "flags": []
      }
    }
  }
}'

echo "PATCHING HOST"
echo "${host_patch}" | jq .

curl -s -X PATCH "${HOST_PROXY_API_PATH}/${host}/status" \
    -H "Content-type: application/merge-patch+json" \
    -d "${host_patch}"

oc get baremetalhost -n openshift-machine-api -o yaml "${host}"

● Confirm members of etcd (3 members)

oc rsh -n openshift-etcd <>
etcdctl member list -w table

● Wait for etcd-operator

oc get clusteroperator etcd

● Confirm health of the etcd

oc rsh -n openshift-etcd <>
etcdctl endpoint health

● Confirm health of the cluster

oc get nodes,co,clusterversion

Step 4: Re-create ODF Member (Storage Resurrection!)

● Create and download a new ISO from the Assisted installer service

● Delete the DNS A, PTR, and DHCP records associated with the old BM

node (which we deleted in the previous steps).

● Boot the old BM node (which we deleted in the previous steps) from

the ISO.

● Create a worker node by Assisted installer service from the BM node

● SSH to the node, and delete the old ODF disks partition

sudo fdisk /dev/<>
Command: d
Partition number:
Partition X has been deleted.
Run w command to write the changes and reboot the system.

In case the output of command above is : there is no partitions, proceed

to the method below:

DISKS=
"/dev/disk/by-id/scsi-360050763808104bc28000000000000eb
/dev/disk/by-id/scsi-360050763808104bc28000000000000ef
/dev/disk/by-id/scsi-360050763808104bc28000000000000f3"

or

DISKS=
"/dev/sdb /dev/sdc /dev/sdd /dev/sde"

for disk in $DISKS; do sgdisk --zap-all $disk;done

● Check the CSRs

oc get csr
oc adm certificate approve <>

● Confirm that the new node is in Ready state.

oc get node

● Define new node variable

export NEW_NODE=<node_name>

● Apply the OpenShift Data Foundation label to the new node

oc label node $NEW_NODE cluster.ocs.openshift.io/openshift-storage=""

● Add a new worker node to the localVolumeDiscovery and

localVolumeSet

oc get -n openshift-local-storage localvolumediscovery
oc edit -n openshift-local-storage localvolumediscovery <auto-discover-devices>

Example output

[...]
nodeSelector:
  nodeSelectorTerms:
    - matchExpressions:
        - key: kubernetes.io/hostname
          operator: In
          values:
            - server1.example.com
            - server2.example.com
            # - (old BM node) server3.example.com
            - newnode.example.com
[...]

● Update the localVolumeSet definition to include the new node, and

remove the failed node

oc get -n openshift-local-storage localvolumeset
oc edit -n openshift-local-storage localvolumeset <localblock>

Example output

[...]
nodeSelector:
  nodeSelectorTerms:
    - matchExpressions:
        - key: kubernetes.io/hostname
          operator: In
          values:
            - server1.example.com
            - server2.example.com
            # - (old BM node) server3.example.com
            - newnode.example.com
[...]

● Verify that the new <localblock> Persistent Volume (PV) is available

oc get pv | grep <localblock> | grep Available
oc get pv | grep Available

Example output

local-pv RWO Delete Available localblock 14s

● Remove the failed OSD from the cluster. You can specify multiple failed OSDs if required.

<failed_osd_id>

The <failed_osd_id> is the integer in the pod name immediately after the “rook-ceph-osd” prefix.

You can add comma-separated OSD IDs in the command to remove more than one OSD.

For example: FAILED_OSD_IDS=0,1,2

oc rsh -n openshift-storage deploy/rook-ceph-tools ceph osd tree

oc process -n openshift-storage ocs-osd-removal \
  -p FAILED_OSD_IDS=<failed_osd_ids> \
  -p FORCE_OSD_REMOVAL=true | oc create -n openshift-storage -f -

● Verify that the OSD was removed successfully by checking the status of

the ocs-osd-removal-job pod. A status of Completed confirms that the

OSD removal job succeeded.

oc get pod -l job-name=ocs-osd-removal-job -n openshift-storage

● Ensure that the OSD removal is completed

oc logs -l job-name=ocs-osd-removal-job -n openshift-storage --tail=-1 | egrep -i
'completed removal'

Example output

cephosd: completed removal of OSD 0

If the ocs-osd-removal-job fails, and the pod is not in the expected

Completed state, check the pod logs for further debugging

For example

oc logs -l job-name=ocs-osd-removal-job -n openshift-storage
--tail=-1

● Identify the Persistent Volume (PV) associated with the Persistent

Volume Claim (PVC)

localblock=$(oc get -n openshift-local-storage localvolumeset -o jsonpath='{.items[0].metadata.name}')
oc get pv -L kubernetes.io/hostname | grep $localblock | grep Released

Example output

local-pv 1490Gi RWO Delete Released
openshift-storage/ocs-deviceset-0-data-0-6c5pw <localblock>
2d22h compute-1

● If there is a PV in Released state, delete it

oc delete pv <persistent volume>

● Identify the crashcollector pod deployment

echo "$NEW_NODE"
oc get deployment -l app=rook-ceph-crashcollector,nodeName="$NEW_NODE" -n openshift-storage

● If there is an existing crashcollector pod deployment, delete it

oc delete deployment -l app=rook-ceph-crashcollector,nodeName=$NEW_NODE -n openshift-storage

● Delete the ocs-osd-removal-job

oc delete -n openshift-storage job ocs-osd-removal-job

● Verify that the new node is present in the output

oc get nodes --show-labels | grep "cluster.ocs.openshift.io/openshift-storage=" | cut -d' ' -f1

● Confirm that at least the following pods on the new node are in Running

state

oc get pods -n openshift-storage -o wide | grep -E 'csi-cephfsplugin-|csi-rbdplugin-' | grep "$NEW_NODE"

● Verify that all other required OpenShift Data Foundation pods are in

Running state

oc get pod -n openshift-storage | grep mon
oc get pods -o wide -n openshift-storage| egrep -i $NEW_NODE | egrep osd

● Make sure that the ODF cluster is in a healthy state

oc rsh -n openshift-storage $(oc get pods -n openshift-storage | grep tools | awk '{print
$1}') ceph -s

Step 5: Architecture Status (Rinse, Repeat, and Rock!)

Cluster status after one full cycle

Just like shampoo instructions (rinse and repeat!), you’ll need to do this three times — each with care, charm, and a dash of patience — to achieve your perfect cluster setup

Step 6: Finalization: Locking Down the Victory!

Almost there! Let’s wrap this up neatly by unscheduling master nodes

oc edit schedulers.config.openshift.io cluster

Set mastersSchedulable to false

apiVersion: config.openshift.io/v1
kind: Scheduler
metadata:
  name: cluster
spec:
  mastersSchedulable: false
  policy: ""
  name: ""
status: {}

or use the following command

oc patch schedulers.config.openshift.io cluster --type=merge -p \ 
'{"spec": {"mastersSchedulable": false}}'

Troubleshooting (Because Stuff Happens!)

Issue: On the new worker node, an error appears:

host will stop trying to register, host is not allowed to perform the requested operation.

Solution:

● Find the Agent CR of the old Master Node

oc get agent.agent-install.openshift.io -n <namespace> -o json | jq -r '.items[] | "\(.metadata.name) \(.spec.hostname)"'

● Delete the Agent of the old Master Node

Issue: Ceph — OSD will not start due a keyring conflict or to it missing in

Solution: Ceph — OSD will not start due a keyring conflict or to it missing in ‘ceph auth list’

Techy Treasure Chest: Handy Resources & Helpful Links

**https://docs.redhat.com/en/documentation/red_hat_openshift_data_foundation/4.14/html/replacing_nodes/openshift_data_foundation_deployed_using_local_storage_devices#replacing-an-operational-node-using-local-storage-devices_bm-upi-failed**

**How to safely reboot an OCS/ODF 4 node — Red Hat Customer Portal**

**Replacing an unhealthy etcd member — Control plane backup and restore**

**How to replace all master nodes in OpenShift Container Platform 4 — Red Hat Customer Portal**

Wrapping Up Our Cluster Adventure!

You’ve successfully navigated the exciting maze of OpenShift cluster redesign! Special shout-out and heartfelt thanks to Almog Elfasy — this article is based on his outstanding work and invaluable contributions.

Now, go forth and cluster confidently — keep it smooth and stylish!

Happy clustering!