Taming IonosCloud: My RKE2 Cluster API Journey from Zero to Scale

Introduction

I’d never touched IonosCloud before. When I set out to build RKE2 clusters via Cluster API, I ran head‑first into sparse docs, half‑baked examples, and a lot of guesswork. Here’s the unvarnished play‑by‑play of how I went from zero to a five‑node worker cluster — plus the questions I’m wrestling with next.

Why IonosCloud + Cluster API + RKE2?

• Cloud-native efficiency: Treat infra as code, spin up clusters on demand.
• RKE2 benefits: Lightweight, opinionated runtime from Rancher. We standardize on RKE2 across all environments and use Rancher for seamless, centralized cluster management.
• Cluster API promise: A unified, declarative way to manage k8s clusters across providers.

However, it’s still early days for the IonosCloud Cluster API provider — its adoption remains limited, RKE2’s examples don’t reference it, and there’s virtually no activity in the Kubernetes Slack #cluster-api-ionoscloud channel.

Initial Roadblocks

1. Zero Ionos experience
2. Provider mismatch

• Ionos docs assume KubeAdmControlPlane & Bootstrap
• RKE2 examples assume cloud providers like AWS, Azure

1. Sparse docs + tiny Quickstart
2. No network examples for private subnets

I had to reverse‑engineer, stitch examples together, and push through failures.

My Step‑by‑Step Plan

Each step had to succeed before moving on — no skipping ahead.

1. One Control Plane Node

• Generated SSH key, API credentials
• Configured Cluster and RKE2ControlPlane with a single replica
• Verified etcd health and API reachability

2. Three Control Plane Nodes

• Adjusted RKE2ControlPlane.spec.replicas to 3
• Handled certificate distribution and HA load balancer setup
• Ensured quorum and failover

3. Add Three Worker Nodes

• Defined RKE2ConfigTemplate with worker role
• Applied MachineDeployment targeting three replicas
• Confirmed node join via kubectl get nodes

Tip: Always check the IonosCloud control‑plane VM flavors — mismatched CPU/RAM kills performance.

One Control Plane Node

First we need to get token from IonosCloud and add it to secret. Even if we add Kubernetes object this secret later will be Owned by Ionos-Cluster-API provider (need to know this if you manage your secrets with some external security tools which takes ownership of secrets, e.g. SopsSecrets)

apiVersion: v1
kind: Secret
metadata:
  name: "ionos-cloud-credentials"
type: Opaque
stringData:
  token: "your-token-from-ionoscloud"

Next we define our Cluster for Cluster API and IonosCloudCluster

---
apiVersion: cluster.x-k8s.io/v1beta1
kind: Cluster
metadata:
  name: "test-cluster"
  labels:
    cluster.x-k8s.io/cluster-name: "test-cluster"
spec:
  clusterNetwork:
    pods:
      cidrBlocks:
      - 10.45.0.0/16
    services:
      cidrBlocks:
      - 10.46.0.0/16
    serviceDomain: cluster.local
  infrastructureRef:
    apiVersion: infrastructure.cluster.x-k8s.io/v1alpha1
    kind: IonosCloudCluster
    name: "test-cluster"
  controlPlaneRef:
    kind: RKE2ControlPlane
    apiVersion: controlplane.cluster.x-k8s.io/v1beta1
    name: "test-control-plane"

For IonosCloudCluster there is prerequisite which cannot be done using provider and it’s reserving FailoverIP in your desired datacenter.

---
apiVersion: infrastructure.cluster.x-k8s.io/v1alpha1
kind: IonosCloudCluster
metadata:
  name: "test-cluster"
spec:
  controlPlaneEndpoint:
    host: $failover_ip
    port: 6443
  location: "de/fra"
  credentialsRef:
    name: "ionos-cloud-credentials"

Now it’s time to define our RKE2ControlPlane. This is working example of RKE2ControlPlane, but let me explain all things defined here, so no questions later on.

First we add some files to our OS for Kubernetes to pickup.

We adjust coredns deployment by appending HelmChartConfig for it. This is required as we need working coredns before node is set ready, as it’s needed for ionos-cloud-control-manager (it resolves api.ionos.com).

Next we add kube-vip deployment, remember i told that we need to Reserve FailoverIP? Kube-vip makes that address alive. Think of this as Keepalived for Kubernetes. Check your OS which you chose has same interface name ens6 (Ubuntu has it as ens6). This should be same as default interface. We also need some ServiceAccount and Role for kube-vip.

Then we need the Ionos Cloud Control Manager, they provide HelmChart for it. Configuration is pretty straight forward.

We alter some sysctl as well, as it will increase our Kubernetes performance.

Next we add static resolv.conf to be used for kubelet, as Ionos by default assigns more then 3 nameservers, and there is limit of 3 in kubelet. (Could be this can fixed by disabling IPV6, as 2 of nameservers there are IPV6) but for test cluster this is fine.

For Kube-vip to work, we need to load some modules, (To be honest i’m not sure they are needed, but as they where in example with Kubeadm i added here as well)

For debugging purposes on control planes nodes it’s nice to have some aliases, so we set for root user .bash_profile (this is up to you what you add there)

In this example i’ve also added ie-csi configfile which would later be picked up by CSI controller, but CSI controller is out of this example scope.

Next comes agentConfig where we set some needed settings:

We set registrationMethod: "control-plane-endpoint" which tells new nodes to register themselves to previously mentioned FailoverIP.

As ClusterAPI depends on node UUID, we need to set this for kubelet on first run, so in postRKE2Commands we add 2 commands which get UUID and patch node with value.

All other not mentioned settings should be understandable, if no drop into comments and we can discuss those.

---
apiVersion: controlplane.cluster.x-k8s.io/v1beta1
kind: RKE2ControlPlane
metadata:
  name: test-control-plane
spec: 
  replicas: 1
  version: v1.32.5+rke2r1
  rolloutStrategy:
    type: "RollingUpdate"
    rollingUpdate:
      maxSurge: 1
  files:
    - path: /var/lib/rancher/rke2/server/manifests/rke2-coredns-config.yaml
      content: |
        apiVersion: helm.cattle.io/v1
        kind: HelmChartConfig
        metadata:
          name: rke2-coredns
          namespace: kube-system
        spec:
          valuesContent: |-
            tolerations:
            - key: "node.cloudprovider.kubernetes.io/uninitialized"
              operator: "Equal"
              effect: "NoSchedule"
              value: "true"
            - effect: NoSchedule
              key: node.kubernetes.io/not-ready
    - path: /var/lib/rancher/rke2/server/manifests/kube-vip.yaml
      content: |
        ---
        apiVersion: v1
        kind: Pod
        metadata:
          name: kube-vip
          namespace: kube-system
        spec:
          serviceAccountName: kube-vip
          containers:
          - args:
            - manager
            env:
            - name: cp_enable
              value: "true"
            - name: vip_interface
              value: "ens6"
            - name: address
              value: "$failover_ip"
            - name: port
              value: "6443"
            - name: vip_arp
              value: "true"
            - name: vip_leaderelection
              value: "true"
            - name: vip_leaseduration
              value: "15"
            - name: vip_renewdeadline
              value: "10"
            - name: vip_retryperiod
              value: "2"
            image: ghcr.io/kube-vip/kube-vip:v0.7.1
            imagePullPolicy: IfNotPresent
            name: kube-vip
            resources: {}
            securityContext:
              capabilities:
                add:
                - NET_ADMIN
                - NET_RAW
            volumeMounts:
            - mountPath: /etc/rancher/rke2/rke2.yaml
              name: kubeconfig
          hostAliases:
          - hostnames:
            - kubernetes
            - localhost
            ip: 127.0.0.1
          hostNetwork: true
          volumes:
          - hostPath:
              path: /etc/rancher/rke2/rke2.yaml
              type: FileOrCreate
            name: kubeconfig
        ---
        apiVersion: v1
        kind: ServiceAccount
        metadata:
          name: kube-vip
          namespace: kube-system
        ---
        apiVersion: rbac.authorization.k8s.io/v1
        kind: Role
        metadata:
          name: leader-locking-kube-vip
          namespace: kube-system
        rules:
        - apiGroups:
          - ""
          resources:
          - configmaps
          verbs:
          - watch
        - apiGroups:
          - ""
          resourceNames:
          - kube-vip
          resources:
          - configmaps
          verbs:
          - get
          - update
        - apiGroups:
          - coordination.k8s.io
          resources:
          - leases
          verbs:
          - create
          - get
          - list
          - update
          - watch
        - apiGroups:
          - coordination.k8s.io
          resources:
          - leasecandidates
          verbs:
          - create
          - get
          - list
          - update
          - watch
        ---
        apiVersion: rbac.authorization.k8s.io/v1
        kind: RoleBinding
        metadata:
          name: leader-locking-kube-vip
          namespace: kube-system
        roleRef:
          apiGroup: rbac.authorization.k8s.io
          kind: Role
          name: leader-locking-kube-vip
        subjects:
        - kind: ServiceAccount
          name: kube-vip
          namespace: kube-system

    - path: /var/lib/rancher/rke2/server/manifests/ionos-ccm.yaml
      content: |
        ---
        apiVersion: helm.cattle.io/v1
        kind: HelmChart
        metadata:
          name: ionoscloud-cloud-controller-manager
          namespace: kube-system
        spec:
          chart: oci://ghcr.io/ionos-cloud/helm-charts/ionoscloud-cloud-controller-manager
          version: 0.1.1
          targetNamespace: kube-system
          bootstrap: true
          valuesContent: |-
            replicaCount: 1
            image:
              repository: ghcr.io/ionos-cloud/ionoscloud-cloud-controller-manager
              pullPolicy: IfNotPresent
              tag: "v1.29.2"
            resources:
              requests:
                cpu: 10m
                memory: 30Mi
              limits:
                cpu: 50m
                memory: 150Mi
            ccm:
              # clusterName must be a unique identifier of the K8s cluster the CCM is running in.
              # It is used in names of automatically reserved IP blocks for load balancers.
              clusterName: asterns-test
              # whether the cluster is private, i.e. whether its nodes have no public IPs
              private: true
              # The CCM uses klog for logging. It has several INFO sublevels:
              # https://github.com/kubernetes/community/blob/9cfd840e1cd9376f562662dfe8135d3042a1e4cd/contributors/devel/sig-instrumentation/logging.md
              klogLevel: 3
              # number of services that are allowed to sync concurrently.
              # Larger number = more responsive service management = more responsive load balancer management
              concurrentServiceSyncs: 3
              terminationGracePeriodSeconds: 120
              # Additional command-line arguments
              extraArgs: {}

            # It is intended to be used for installing ionoscloud-cloud-controller-manager to cluster-api based clusters.
            cloudConfig:
              # Set externalSecret to the name of an externally managed secret if you do not want the helm chart to create the cloud-config secret.
              # Either externalSecret or token and datacenters must be specified.
              externalSecret: ""
              # REMEMBER: IONOS tokens have a TTL and must be refreshed.
              token: "$token"
              # A list of datacenter IDs. Every datacenter containing nodes must be given here.
              datacenters:
                - "$datacenter_id"

            metrics:
              # If true, enables metrics scraping.
              enabled: false

    - path: /etc/sysctl.d/k8s.conf
      content: |
        fs.inotify.max_user_watches = 65536
        net.netfilter.nf_conntrack_max = 1000000
    - path: /etc/resolv-static.conf
      permissions: '0644'
      content: |
        nameserver 1.1.1.1
        nameserver 8.8.8.8
    - path: /etc/modules-load.d/k8s.conf
      content: |
        ip_vs
        ip_vs_rr
        ip_vs_wrr
        ip_vs_sh
        ip_vs_sed
    - path: /root/.bash_profile
      permissions: '0644'
      content: |
        #!/usr/bin/env bash
        PATH=${PATH}:/opt/rke2/bin:/opt/bin

        kubectl() {
          /var/lib/rancher/rke2/bin/kubectl $@ --kubeconfig /etc/rancher/rke2/rke2.yaml
        }

        ns() {
          kubectl config set-context --current --namespace=${1:-kube-system}
        }

        logs() {
          svcs=("rke2-server" "rancher-system-agent" "kubelet")
          PS3="Choose a service: "
          select svc in "${svcs[@]}"; do
            case $svc in
              "rke2-server")
                journalctl -u rke2-server -f --no-pager
                ;;
              "rancher-system-agent")
                journalctl -u rancher-system-agent -f --no-pager
                ;;
              "kubelet")
                tail -f /var/lib/rancher/rke2/agent/logs/kubelet.log
                ;;
              *)
                continue
                ;;
            esac
            break
          done
        }

        alias k='kubectl'
        alias kgp='k get pods'
        alias kd='k describe'
        alias kdp='kd pod'
        alias kgd='k get deploy'
        alias kgs='k get services'
        alias kgn='k get nodes'
        alias klf='k logs -f'
        alias klp='k logs --previous'
    - content: |
        {
          "datacenter-id": "$UUID"
        }
      owner: root:root
      path: /etc/ie-csi/cfg.json
      permissions: '0644'
  agentConfig: 
    format: cloud-config
    airGapped: false
    additionalUserData:
      config: |
        users:
          - name: root
            sudo: ALL=(ALL) NOPASSWD:ALL
            ssh-authorized-keys:
              - 'ssh-ed25519 .....'
    kubelet:
      extraArgs:
        - --cloud-provider=external
        - --resolv-conf=/etc/resolv-static.conf
  registrationMethod: "control-plane-endpoint"
  preRKE2Commands:
    - sysctl --system
  postRKE2Commands:
    - export system_uuid=$(/var/lib/rancher/rke2/bin/kubectl --kubeconfig /etc/rancher/rke2/rke2.yaml get node $(hostname) -ojsonpath='{..systemUUID }')
    - >
      /var/lib/rancher/rke2/bin/kubectl --kubeconfig /etc/rancher/rke2/rke2.yaml patch node $(hostname) --type strategic -p '{"spec": {"providerID": "ionos://'${system_uuid}'"}}'
  serverConfig:
    disableComponents:
      pluginComponents:
        - rke2-ingress-nginx
        - rke2-metrics-server
      kubernetesComponents:
        - cloudController
    kubeAPIServer:
      extraArgs:
      - --anonymous-auth=true
    cloudProviderName: external
    cni: cilium
    etcd: {}
  machineTemplate:
    infrastructureRef:
      apiVersion: infrastructure.cluster.x-k8s.io/v1beta1
      kind: IonosCloudMachineTemplate
      name: test-control-plane
    nodeDrainTimeout: 2m
    nodeDeletionTimeout: 30s
    nodeVolumeDetachTimeout: 5m

Now we need IonosCloudMachineTemplate You define specification of the server you want to create.

---
apiVersion: infrastructure.cluster.x-k8s.io/v1alpha1
kind: IonosCloudMachineTemplate
metadata:
  name: test-control-plane
spec:
  template:
    spec:
      datacenterID: $datacenter_uuid
      disk:
        image:
          id: $image_id
      memoryMB: 8192
      numCores: 4
      type: VCPU

That’s it, apply these manifests and you will have 1 node Kubernetes ControlPlane. Next step increase replicas: 3 and wait until your 3 ControlPlanes are up.

N.B. If your 3 replicas do not come up, check Security Group rules in IonosCloud. There are quite few ports which needs access over public ip. (I will leave for you to find out, which ports are needed there)

You can access your Kubernetes API by saving test-kubeconfig secret into file and pointing your kubectl to it.

kubectl get secret test-kubeconfig -o json | jq -r .data.value | base64 -d > kubeconfig.yaml

Next access your Kubernetes API:

kubectl get nodes --kubeconfig kubeconfig.yaml
NAME                       STATUS   ROLES                       AGE     VERSION
test-control-plane-kmm8d   Ready    control-plane,etcd,master   4h6m    v1.32.5+rke2r1
test-control-plane-wjzrl   Ready    control-plane,etcd,master   3h59m   v1.32.5+rke2r1
test-control-plane-zfv2w   Ready    control-plane,etcd,master   4h2m    v1.32.5+rke2r1

What’s Next?

Now we have working ControlPlane nodes we need some worker nodes, let’s see what we can do, this requires a bit different inputs.

Our Nodes will be living in MachineDeployment

---
apiVersion: cluster.x-k8s.io/v1beta1
kind: MachineDeployment
metadata:
  labels:
    cluster.x-k8s.io/cluster-name: test-cluster
  name: test-workers
spec:
  clusterName: test-cluster
  replicas: 3
  selector:
    matchLabels: 
      cluster.x-k8s.io/cluster-name: test-cluster
  template:
    metadata:
      labels:
        cluster.x-k8s.io/cluster-name: test-cluster
        node-role.kubernetes.io/node: ""
    spec:
      bootstrap:
        configRef:
          apiVersion: bootstrap.cluster.x-k8s.io/v1beta1
          kind: RKE2ConfigTemplate
          name: test-workers
      clusterName: asterns-test
      infrastructureRef:
        apiVersion: infrastructure.cluster.x-k8s.io/v1alpha1
        kind: IonosCloudMachineTemplate
        name: test-workers
      version: v1.32.5+rke2r1

Next we add RKE2ConfigTemplate which will define how to configure nodes on startup.

---
apiVersion: bootstrap.cluster.x-k8s.io/v1beta1
kind: RKE2ConfigTemplate
metadata:
  name: test-workers
spec:
  template:
    spec:
      files:
      - path: /etc/sysctl.d/k8s.conf
        content: |
          fs.inotify.max_user_watches = 65536
          net.netfilter.nf_conntrack_max = 1000000
      - path: /etc/resolv-static.conf
        permissions: '0644'
        content: |
          nameserver 1.1.1.1
          nameserver 8.8.8.8
      - content: |
          {
            "datacenter-id": "$datacenter_id"
          }
        owner: root:root
        path: /etc/ie-csi/cfg.json
        permissions: '0644'
      postRKE2Commands:
        - export system_uuid=$(/var/lib/rancher/rke2/bin/kubectl --kubeconfig /etc/rancher/rke2/rke2.yaml get node $(hostname) -ojsonpath='{..systemUUID }')
        - >
          /var/lib/rancher/rke2/bin/kubectl --kubeconfig /etc/rancher/rke2/rke2.yaml patch node $(hostname) --type strategic -p '{"spec": {"providerID": "ionos://'${system_uuid}'"}}'
      agentConfig:
        additionalUserData:
          config: |
            users:
              - name: root
                sudo: ALL=(ALL) NOPASSWD:ALL
                ssh-authorized-keys:
                  - 'ssh-ed25519 .....'
        kubelet:
          extraArgs:
            - "--cloud-provider=external"

And latest peace of the puzzle is IonosCloudMachineTemplate Which for me is same as for ControlPlanes.

---
apiVersion: infrastructure.cluster.x-k8s.io/v1alpha1
kind: IonosCloudMachineTemplate
metadata:
  name: test-workers
spec:
  template:
    spec:
      datacenterID: $datacenter_id
      disk:
        image:
          id: $image_id
      memoryMB: 4096
      numCores: 2
      type: VCPU

That’s it, you have working cluster with 3 nodes. Deploy your stuff.

Key Takeaways

• Docs ≠ Reality: Expect to fill the gaps yourself.
• Iterate in Small Batches: One replica → three → scale.
• Log Everything: Control‑plane, bootstrap, cloud‑provider API calls.
• Automate Re‑runs: Write scripts for reprovisioning to cut debugging time.

Conclusion

Building RKE2 clusters on IonosCloud via Cluster API isn’t turnkey — yet. But with persistence, you can turn those minimal Quickstarts into production‑ready, HA setups.

Happy Deploying! P.S. I’m not affiliated with IonosCloud.