Integrating Harvester with Rancher

TL;DR: Integrating Harvester with Rancher was trickier than expected. Here’s what went wrong and how I fixed it.

The Setup

I’m running Harvester as my HCI platform, with Rancher v2.13.3 deployed on an RKE2 cluster inside a VM on Harvester itself. DNS is handled by an internal BIND server pointing rancher.dummy.com to the RKE2 VM IP and cert-manager is managing Rancher's TLS certs.

First Blocker — Harvester Can’t Find Rancher

The very first thing that hit me was Harvester nodes not being able to resolve rancher.dummy.com. Makes sense, Harvester was using the corporate DNS servers (10.x.x.x., 10.x.x.x) which had no idea about my internal domain.

To add your internal BIND server to Harvester nodes, follow the official docs link

Once DNS is sorted, verify its working from any Harvester node

The CoreDNS restart matters, it picks up the new /etc/resolve.conf so pods inside the cluster also get the updated DNS. One fix, two layers covered.

If you just want to test quickly without touching DNS, you can drop an /etc/hosts entry on each Harvester node. Once DNS was sorted, curl https://rancher.dummy.com/ping returned pong from the Harvester node. First win.

The Cert Rabbit Hole

This is where the real pain started. The cattle-cluster-agent the pod running in the cattle-system namespace on Harvester that is responsible for maintaining the connection between Harvester and Rancher, syncing cluster state and relaying management instructions crashed immediately after the integration was triggered

The first error was:

unable to read CA file from /etc/kubernetes/ssl/certs/serverca: open /etc/kubernetes/ssl/certs/serverca: no such file or directory Strict CA verification is enabled but encountered error finding root CA

The agent starts up and the very first thing it does is look for a trusted CA file on disk at /etc/kubernetes/ssl/certs/serverca. Think of this like a security guard checking his list of trusted ID issuers before letting anyone in. The file wasn’t there so the guard had no list. It logged the error but kept going, hoping to find the CA another way.

The second way it tries is by fetching the cacerts setting from Rancher itself at **https://rancher.dummy.com/v3/settings/cacerts**. But before it can trust that response, it needs to verify Rancher’s TLS certificate first. And that’s where everything fell apart:

Certificate chain is not complete, please check if all needed intermediate certificates are included in the server certificate (in the correct order) and if the cacerts setting in Rancher either contains the correct CA certificate (in the case of using self signed certificates) or is empty (in the case of using a certificate signed by a recognized CA). error: tls: failed to verify certificate: x509: certificate signed by unknown authority

How a Certificate Chain Actually Works

Think of it like a chain of trust with three links:

Root CA (the trusted authority) → Intermediate CA (optional, signs on behalf of root) → Leaf cert (the actual server certificate — rancher.dummy.com)

When an agent connects to a server, it receives the leaf cert. It then asks: “Who signed this?” It walks up the chain looking for a signer it recognises. If it reaches a CA it already trusts — job done, connection allowed. If it hits a dead end and finds nobody it recognises — rejected.

For this to work, two things must be true. First, the server must serve the full chain — leaf cert plus all the CA certs above it. Second, the client must have the root CA in its trusted store.

What Was Actually Happening

Rancher had been installed with a SelfSigned ClusterIssuer in cert-manager. Here’s the problem with that:

rancher-selfsigned (ClusterIssuer) → Self-signed — generates a cert that signs ITSELF → tls-rancher-ingress cert → Issuer: empty — nobody signed it → CA: FALSE — it’s not a CA → No CA bundle — nothing for anyone to verify against

A self-signed ClusterIssuer doesn’t create a CA — it just stamps a certificate with its own signature and calls it done. There’s no parent authority, no chain, no bundle. It’s like printing your own ID card and signing it yourself. The security guard takes one look and says: “I don’t know who issued this — you’re not getting in.”

On top of that, nginx was serving its built-in fallback cert — the infamous Acme Co / Kubernetes Ingress Controller Fake Certificate — because the real TLS secret was broken. So the agent wasn’t even seeing a Rancher cert at all.

Running an openssl check confirmed it:

openssl s_client -connect rancher.dummy.com:443 -showcerts </dev/null 2>/dev/null \
  | openssl x509 -noout -text | grep -E "Issuer|CA"

# Issuer: O=Acme Co, CN=Kubernetes Ingress Controller Fake Certificate
# CA:FALSE

That “Acme Co” cert is nginx’s built-in fallback.

The Fix — Building a Real Chain

The solution was to build a proper CA-backed chain from scratch, with a real CA cert at the top that everyone could be told to trust.

First, a genuine CA certificate was created , one with isCA: true — using the existing self-signed issuer just as a one-time bootstrap:

kubectl apply -f - <<EOF
apiVersion: cert-manager.io/v1
kind: Certificate
metadata:
  name: rancher-selfsigned-ca
  namespace: cattle-system
spec:
  isCA: true
  commonName: rancher-selfsigned-ca
  secretName: rancher-selfsigned-ca-secret
  issuerRef:
    name: rancher-selfsigned
    kind: ClusterIssuer
EOF

Then came a gotcha — ClusterIssuers look for their backing secret in the cert-manager namespace, not wherever the certificate lives. Had to copy it across:

kubectl get secret rancher-selfsigned-ca-secret -n cattle-system -o yaml \
  | sed 's/namespace: cattle-system/namespace: cert-manager/' \
  | kubectl apply -f -

With the secret in the right place, a proper CA-backed ClusterIssuer was created and the Rancher ingress annotation updated to use it:

kubectl apply -f - <<EOF
apiVersion: cert-manager.io/v1
kind: ClusterIssuer
metadata:
  name: rancher-ca-clusterissuer
spec:
  ca:
    secretName: rancher-selfsigned-ca-secret
EOF

kubectl edit ingress rancher -n cattle-system
# cert-manager.io/cluster-issuer: rancher-selfsigned  →  rancher-ca-clusterissuer

Deleting the old TLS secret forced cert-manager to regenerate it using the new issuer. But even then, the chain was still incomplete — cert-manager only puts the leaf cert in the secret, not the CA. Had to stitch them together manually:

kubectl get secret tls-rancher-ingress -n cattle-system \
  -o jsonpath='{.data.tls\.crt}' | base64 -d > /tmp/leaf.crt

kubectl get secret rancher-selfsigned-ca-secret -n cert-manager \
  -o jsonpath='{.data.tls\.crt}' | base64 -d > /tmp/ca.crt

cat /tmp/leaf.crt /tmp/ca.crt > /tmp/fullchain.crt

kubectl create secret tls tls-rancher-ingress \
  -n cattle-system \
  --cert=/tmp/fullchain.crt \
  --key=/tmp/rancher.key \
  --dry-run=client -o yaml | kubectl apply -f -

kubectl rollout restart daemonset/rke2-ingress-nginx-controller -n kube-system

After the nginx restart, openssl finally showed two certificates in the chain and Issuer: CN=rancher-selfsigned-ca. Progress.

The cacerts and Checksum Problem

With the cert chain fixed, the agent was now able to see the correct issuer but still failing. The cacerts setting in Rancher was empty, so the agent had no CA to trust against.

Even with the chain being served correctly by the ingress, the agent also checks Rancher’s cacerts API setting as its source of truth for what CA to trust. Think of it as Rancher saying: “Here is the CA cert I use — you should trust this.” If it’s empty, the agent has nothing to verify against and rejects the connection.

Setting it sounds simple. It isn’t, because the cert has newlines and JSON doesn’t like those. After a few failed attempts with raw kubectl patch, jq turned out to be the clean solution:

CERT=$(cat /tmp/ca.crt | jq -Rs .)
kubectl patch settings.management.cattle.io cacerts \
  --type=merge \
  -p "{\"value\": $CERT}"

Then came the final error — a checksum mismatch:

ERROR: Configured cacerts checksum (9bb…) does not match given — ca-checksum (f040beb2…)

The agent deployment carries a CATTLE_CA_CHECKSUM environment variable — a sha256 fingerprint of the expected CA cert. The checksum in the deployment was out of sync with the cert now in cacerts. Computing the sha256 of the CA cert and updating CATTLE_CA_CHECKSUM in the agent deployment to match was the final piece:

cat /tmp/ca.crt | sha256sum | awk '{print $1}'

kubectl edit deployment cattle-cluster-agent -n cattle-system
# Set CATTLE_CA_CHECKSUM to the sha256 output

Wiring It All Together

With DNS resolving, the cert chain complete, cacerts set and the checksum matching, the actual integration from the UI was anticlimactic in a good way.

From Rancher:

Virtualization Management → Import Existing → Harvester generates a registration URL.

On the Harvester side, paste the CA cert into Settings → additional-ca

and the registration URL into Settings → cluster-registration-url.

Rancher UI shows harvester-cluster → State: Active.

References

• Harvester — Update DNS Server Configuration
• Harvester — Rancher Integration
• Harvester — Virtualization Management