Longhorn: Upgrade

In this article we will show how to upgrade Longhorn storage provisioner.

Preamble

Normally, I do not upgrade the installed Longhorn version. But I got a high memory usage from instance manager pods.

instance-manager pod usage from metrics server

This is unreasonably high. A quick Google search revealed that some versions of Longhorn have memory leaks often caused by Go runtime heap leaks, repeated gRPC connection retries for unreachable endpoints, or unconstrained cache accumulation. More detailed information can be found in the release known issue page. The version I was running is v1.11.0.

Here, we upgrade it to v1.12.0. The data engine is v1. In this particular upgrade, the live Longhorn engine upgrade is supported. Upgrades from other versions might differ from the steps shown here.

Longhorn Upgrade

Longhorn supports a step-by-step upgrade path, upgrading one minor version at a time starting with v1.5.0. The official document on the upgrade can be found here. According to the document, the downgrades are prevented and reverting to the previously installed version is supported without service interruption or downtime.

Pre-Upgrade Checklist

Before the upgrade, check the status of Longhorn volumes.

kubectl get volumes.longhorn.io -n longhorn-system \
    -o custom-columns="NAME:.metadata.name,STATE:.status.state,ROBUSTNESS:.status.robustness"

Make sure that there are no degraded volumes.

Also, verify that all the pods are in Running/Ready status.

kubectl -n longhorn-system get pods

If data loss is not an option, create a backup before the upgrade.

Longhorn (Manager) Upgrade Using Helm

We assume that Longhorn was initially installed using Helm. The initial deployment steps can be found in my previous Longhorn article.

• Longhorn: Distributed Block Storage on Kubernetes

We start by updating Longhorn Helm repo.

# Add Longhorn Helm repo if it is not done already.
helm repo add longhorn https://charts.longhorn.io
helm repo update longhorn
# Find a target upgrade version
helm search repo longhorn --versions

Once we decide on a new version for the upgrade, we can download the helm chart.

helm pull longhorn/longhorn --version 1.12.0 --untar

Now, we are ready to run Helm upgrade.

helm upgrade longhorn ./longhorn --namespace longhorn-system -f override.yaml

We are using the original override.yaml file. This will upgrade Longhorn manager. Once it is done, we should see image tag v1.12.0 from the following command.

# check longhorn-manager and longhorn share manager version.
kubectl -n longhorn-system get pods -lapp=longhorn-manager \
    -ojsonpath="{.items[*].spec}" | jq -r '.containers[].image'

Longhorn Engine Upgrade Using UI

From the top-level menu, click Advanced>Engine Images and check that the new engine image is available.

Longhorn UI (Volumes)

Once it is confirmed, on the Volumes tab (shown above) we will upgrade each attached engine to the new engine using Operation>Upgrade Engine.

Upgrade Engine Image Popup (the new engine is chosen)

Once all the PVCs are switched to the new engine image, the reference count of the old engine image should be zero. You can manually delete the old engine image CRD or wait for it to be deleted by manager. Once the migration is done, you should see only one updated engine image.

# check Longhorn engine image CRDs. 
# To delete the old engine image, the reference count should be zero.
kubectl -n longhorn-system get engineimages.longhorn.io

Now, you can double-check the container image version of the new instance manager.

kubectl -n longhorn-system get pods -llonghorn.io/component=instance-manager \
    -ojsonpath="{.items[*].spec}" | jq -r '.containers[].image'

This completes the Longhorn upgrade process.

More Articles

• On AI
• On Kubernetes
• On Ceph