k8s in air-gapped environment

Scenario

You want to install a k8s application in an air-gapped virtual machine. For this scenario, I will use Ubuntu 24.04.3 LTS as the operating system for the virtual machine. I want to get hashicorp/http-echo up and running with k3s and also configure an ingress for it.

Prerequisites

You need a virtual machine with Ubuntu 24.04.3 LTS, or something very similar. Using other distribution might result in having to modify some commands, especially related to installation of packets or using system services. If you use an Ubuntu distribution, you can call

lsb_release -a

to find out the version of your distribution. This can output something like:

Distributor ID: Ubuntu
Description: Ubuntu 24.04.3 LTS
Release: 24.04
Codename: noble

I will assume you have ssh access to the virtual machine during the setup, but if you do not, you will have to manually do what happens in some of the scripts.

Plan

I will use a few steps to accomplish the setup.

In the first step the goal is to obtain all files that are needed for the setup. Some have to be downloaded, others have to be prepared, and some of them need extra configuration.

The purpose of the second step is to prepare some directory structure on the virtual machine and transfer all files. If you do not have ssh access you need to do this in some other way.

The third step is the setup of k8s. We will be using the k3s distribution.

The fourth step is the setup of our hashicorp/http-echo application.

All scripts that we use must be idempotent, so that we do not break anything if we accidentally run one of them more times. This also makes things easier during development. This can be achieved by adding checks before most commands.

Step1

In this step, I will first create a script to download all necessary files. It starts with

#!/bin/sh

# Exit immediately if a simple command exits with a nonzero exit value
set -e

Let’s add a section to download k3s and related components for an air-gapped environment. Some k3s official documentation related to this can be found at https://docs.k3s.io/installation/airgap.

K3S_VERSION="1.33.3"
K3S_IMAGES_VERSION="1.33.1"
mkdir -p k3s
cd k3s
# Download k3s airgap images
if [ ! -f "k3s-airgap-images-amd64.tar.zst" ]; then
    curl -L -o k3s-airgap-images-amd64.tar.zst "https://github.com/k3s-io/k3s/releases/download/v${K3S_IMAGES_VERSION}%2Bk3s1/k3s-airgap-images-amd64.tar.zst"
else
    echo "k3s images ${K3S_IMAGES_VERSION} already downloaded."
fi
# Download k3s binary
if [ ! -f "k3s" ]; then
    curl -L -o k3s "https://github.com/k3s-io/k3s/releases/download/v${K3S_VERSION}%2Bk3s1/k3s"
else
    echo "k3s ${K3S_VERSION} already downloaded."
fi
# Download k3s install script
if [ ! -f "install.sh" ]; then
    curl -Lo install.sh https://get.k3s.io
else
    echo "k3s install script already downloaded."
fi
cd ..

This code snippet downloads the k3s binary, its necessary container images for an air-gapped installation and its installation script. If they were already downloaded, they will be skipped next time.

Let’s download helm next. While it is not needed for hashicorp/http-echo, you will probably need it for your own application.

HELM_VERSION="3.15.1"
mkdir -p helm
cd helm
if [ ! -f "helm-linux-amd64.tar.gz" ]; then
    wget -O helm-linux-amd64.tar.gz https://get.helm.sh/helm-v${HELM_VERSION}-linux-amd64.tar.gz
else
    echo "Helm already downloaded."
fi
cd ..

Since we want to configure ingress, we have to download all necessary components for it. I will go with ingress-nginx because of it’s features. Since I want to use the LoadBalancer ingress type, we will need a load balancer implementation. I went with MetalLB.

METALLB_VERSION="0.14.5"
INGRESS_NGINX_VERSION="1.10.0"
INGRESS_NGINX_KUBE_WEBHOOK_VERSION="1.4.0"
mkdir k8s-infrastructure
cd k8s-infrastructure
# Download MetalLB YAML
if [ ! -f "metallb.yaml" ]; then
    curl -L -o metallb.yaml https://raw.githubusercontent.com/metallb/metallb/v${METALLB_VERSION}/config/manifests/metallb-native.yaml
else
    echo "Metallb YAML already downloaded."
fi
# Download Ingress-Nginx YAML
if [ ! -f "ingress-nginx.yaml" ]; then
    curl -L -o ingress-nginx.yaml https://raw.githubusercontent.com/kubernetes/ingress-nginx/controller-v${INGRESS_NGINX_VERSION}/deploy/static/provider/baremetal/deploy.yaml
    sed -i 's/type: NodePort/type: LoadBalancer/' ingress-nginx.yaml
else
    echo "Ingress-Nginx YAML already downloaded."
fi
# Download MetalLB and Ingress-Nginx images
if [ ! -f "metallb-nginx-images.tar" ]; then
    docker pull --platform linux/amd64 quay.io/metallb/controller:v${METALLB_VERSION}
    docker pull --platform linux/amd64 quay.io/metallb/speaker:v${METALLB_VERSION}
    docker pull --platform linux/amd64 registry.k8s.io/ingress-nginx/controller:v${INGRESS_NGINX_VERSION}
    docker pull --platform linux/amd64 registry.k8s.io/ingress-nginx/kube-webhook-certgen:v${INGRESS_NGINX_KUBE_WEBHOOK_VERSION}
    docker save quay.io/metallb/controller:v${METALLB_VERSION} -o metallb-controller.tar
    docker save quay.io/metallb/speaker:v${METALLB_VERSION} -o metallb-speaker.tar
    docker save registry.k8s.io/ingress-nginx/controller:v${INGRESS_NGINX_VERSION} -o nginx-controller.tar
    docker save registry.k8s.io/ingress-nginx/kube-webhook-certgen:v${INGRESS_NGINX_KUBE_WEBHOOK_VERSION} -o nginx-certgen.tar
    tar -cvf metallb-nginx-images.tar *.tar
else
    echo "MetalLB and Ingress-Nginx images already downloaded."
fi
cd ..

The first part on this code snippet is the download of MetalLB yaml file, which is pretty straightforward. The second part downloads the ingress-nginx yaml file; we need to change its type to LoadBalancer since the bare metal variant used NodePort as default. The third part is to download all needed container images for MetalLB and ingress-nginx. If you use another version of ingress-nginx, make sure to check its yaml file for the rest of the dependencies.

We are not done yet with this part. MetalLB needs an ip pool. Let’s prepare a first draft for it:

apiVersion: metallb.io/v1beta1
kind: IPAddressPool
metadata:
  name: default-pool
  namespace: metallb-system
spec:
  addresses:
  - 1.2.3.100-1.2.3.120
---
apiVersion: metallb.io/v1beta1
kind: L2Advertisement
metadata:
  name: default-advertisement
  namespace: metallb-system
spec:
  ipAddressPools:
  - default-pool

Now you have to make a change specific to your virtual machine. You need to change the addresses 1.2.3.100–1.2.3.120 with values that work for your virtual machine. You can call

ip addr show ens18 | awk '/inet / && !/127.0.0.1/ {print $2}'

to get your IP and your the network size, which determines your available IP range. This can look like:

1.2.3.4/22

, where 1.2.3.4 is your IP and /22 is your the network size. Depending on the network size, you can choose some IPs for the IP pool. Higher numbers, like greather than 100 are usually safe from DHCP. You also want to avoid a range that includes your virtual machine’s IP, therefore in our case a range of 1.2.3.100–1.2.3.120 is a good choice; note that it does not include the virtual machine’s IP 1.2.3.4. This is a good part to ask someone or some AI if you are not confident :).

Let’s also download the container images for hashicorp/http-echo:

mkdir -p testapp
cd testapp
if [ ! -f "testapp.tar" ]; then
    docker pull --platform linux/amd64 hashicorp/http-echo:latest
    docker save hashicorp/http-echo:latest -o testapp.tar
else
    echo "Testing app images already downloaded."
fi
cd ..

Let’s prepare the yaml files also for our testing application. We can start with the namespace:

apiVersion: v1
kind: Namespace
metadata:
  name: testapp

Let’s add a deployment:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: testapp
  namespace: testapp
spec:
  replicas: 1
  selector:
    matchLabels:
      app: testapp
  template:
    metadata:
      labels:
        app: testapp
    spec:
      containers:
      - name: testapp
        image: hashicorp/http-echo:latest
        imagePullPolicy: Never
        args: ["-text='Test App is Working!'"]
        ports:
        - containerPort: 5678

Using hashicorp/http-echo we can give a custom text as an argument for some extra customization. Let’s also add a service:

apiVersion: v1
kind: Service
metadata:
  name: testapp-service
  namespace: testapp
spec:
  type: LoadBalancer
  selector:
    app: testapp
  ports:
  - port: 80
    targetPort: 5678

Finally let’s add ingress:

apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: testapp-ingress
  namespace: testapp
spec:
  ingressClassName: nginx
  rules:
  - host: testapp.mycompany.com
    http:
      paths:
      - path: /
        pathType: Prefix
        backend:
          service:
            name: testapp-service
            port:
              number: 80

I assume that DNS is configured and testapp.mycompany.com is pointing to your virtual machine. You will have to configure and change this with your custom value.

We can now run

download.sh

to download all needed files.

Step2

Here we want to transfer all prepared files to the virtual machine. If you do not have ssh access to the virtual machine, you need to create the directory structure and transfer the files some other way. If we go with a script, we need to set up some environment variables:

• SSH_USER: user to connect to the virtual machine
• SSH_HOST: IP of the virtual machine
• WORK_DIR: some location on the virtual machine where we can do our stuff. This should be some location with a lot of disk space, since your custom application could be much bigger, and you might run out of disk space, if you do not keep this in mind.

To find out more information about disk configuration, you can use the following commands:

df -h
lsblk -f
mount

In many cases the location with a lot of space is a disk that might or might not be mounted. If it is not mounted, you will need to do that also. The

lsblk -f

will provide you with the UUID of the disk, which you will need later. First check if there is a filesystem on the disk with

sudo file -s /dev/sdb1

You can call

sudo mkdir -p /mnt/mydata
sudo mount /dev/sdb1 /mnt/mydata

for example. The disk might be somewhere else in your case. You should make the mount persistent by adding it to fstab:

echo "UUID=12345678-ABCD-ABCD-ABCD-123456789012 /mnt/mydata ext4 defaults 0 2" | sudo tee -a /etc/fstab

Here is the location where you need the UUID which you obtained earlier. Consider changing the owner if needed:

sudo chown -R myuser:myuser /mnt/mydata

All of this might not be needed if it is already set up on your virtual machine.

Let’s now get to the transfer script.

Let’s create the directory structure:

ssh ${SSH_USER}@${SSH_HOST} "
    mkdir -p ${WORK_DIR}/temp
    mkdir -p ${WORK_DIR}/k3s
    mkdir -p ${WORK_DIR}/k8s-infrastructure
    mkdir -p ${WORK_DIR}/helm
    mkdir -p ${WORK_DIR}/scripts
    mkdir -p ${WORK_DIR}/testapp
"
if [ $? -ne 0 ]; then
    echo "Failed to create directories on remote machine"
    exit 1
fi

Now we want to transfer the files in an idempotent way. We need to write a custom function for this, since there is not command that skips the transfer, if the file already exists at the destination:

transferIfNeeded() {
    local LOCAL_FILE="$1"
    local REMOTE_PATH="$2"
    local FILENAME=$(basename "${LOCAL_FILE}")
    local REMOTE_FILE="${REMOTE_PATH}/${FILENAME}"
    echo "Checking: ${LOCAL_FILE} -> ${REMOTE_FILE}"
    if ssh ${SSH_USER}@${SSH_HOST} "[ -f '${REMOTE_FILE}' ]"; then
        echo "  Remote file already exists, skipping transfer."
        return 0
    else
        echo "  Remote file doesn't exist, transferring..."
        scp -p "${LOCAL_FILE}" ${SSH_USER}@${SSH_HOST}:"${REMOTE_FILE}"
        if [ $? -eq 0 ]; then
            echo "  Transfer completed successfully."
            return 0
        else
            echo "  Transfer failed!"
            return 1
        fi
    fi
}

Now we can transfer all the files to the desired location by using this function:

transferIfNeeded "k3s/k3s" "${WORK_DIR}/k3s"
transferIfNeeded "k3s/install.sh" "${WORK_DIR}/k3s"
transferIfNeeded "k3s/k3s-airgap-images-amd64.tar.zst" "${WORK_DIR}/k3s"

transferIfNeeded "k8s-infrastructure/metallb.yaml" "${WORK_DIR}/k8s-infrastructure"
transferIfNeeded "k8s-infrastructure/ingress-nginx.yaml" "${WORK_DIR}/k8s-infrastructure"
transferIfNeeded "k8s-infrastructure/metallb-ip-pool.yaml" "${WORK_DIR}/k8s-infrastructure"
transferIfNeeded "k8s-infrastructure/metallb-nginx-images.tar" "${WORK_DIR}/k8s-infrastructure"

transferIfNeeded "helm/helm-linux-amd64.tar.gz" "${WORK_DIR}/helm"

transferIfNeeded "scripts/setupk8s.sh" "${WORK_DIR}/scripts"
transferIfNeeded "scripts/setupTestApp.sh" "${WORK_DIR}/scripts"

transferIfNeeded "testapp/testapp.tar" "${WORK_DIR}/testapp/images"
transferIfNeeded "testapp/namespace.yaml" "${WORK_DIR}/testapp/yamls"
transferIfNeeded "testapp/deployment.yaml" "${WORK_DIR}/testapp/yamls"
transferIfNeeded "testapp/service.yaml" "${WORK_DIR}/testapp/yamls"
transferIfNeeded "testapp/ingress.yaml" "${WORK_DIR}/testapp/yamls"

If you are in the development phase, and you actually want to overwrite something, you can do that using scp, for example:

scp -p "scripts/setupk8s.sh" ${SSH_USER}@${SSH_HOST}:"${WORK_DIR}/scripts/setupk8s.sh"

We can now run

SSH_USER=myuser SSH_HOST=1.2.3.4 WORK_DIR=/mnt/mydata sh transfer.sh

to transfer all needed files to the virtual machine.

Step3

Now we want to configure k8s on the virtual machine. We are going to use the previous WORK_DIR environment variable throughout the script.

Before installing k3s, we must change the place where container images are stored, since we can easily run out of disk space with the default configuration. We want to use some location related to WORK_DIR, where we know there is a lot of disk space available.

Since we know we will be using k3s, we can start by creating some directories for the source of the container images and the target location, somewhere in WORK_DIR:

sudo mkdir -p ${WORK_DIR}/containerd-storage/io.containerd.snapshotter.v1.overlayfs
sudo mkdir -p ${WORK_DIR}/containerd-storage/io.containerd.content.v1.content
sudo mkdir -p /var/lib/rancher/k3s/agent/containerd/io.containerd.snapshotter.v1.overlayfs
sudo mkdir -p /var/lib/rancher/k3s/agent/containerd/io.containerd.content.v1.content

A good way to do the redirect is to use mounts:

if ! mountpoint -q /var/lib/rancher/k3s/agent/containerd/io.containerd.snapshotter.v1.overlayfs; then
    sudo mount --bind ${WORK_DIR}/containerd-storage/io.containerd.snapshotter.v1.overlayfs /var/lib/rancher/k3s/agent/containerd/io.containerd.snapshotter.v1.overlayfs
fi
if ! mountpoint -q /var/lib/rancher/k3s/agent/containerd/io.containerd.content.v1.content; then
    sudo mount --bind ${WORK_DIR}/containerd-storage/io.containerd.content.v1.content /var/lib/rancher/k3s/agent/containerd/io.containerd.content.v1.content
fi

The last step is to make them persistent by adding them to fstab:

if ! grep -q "containerd.snapshotter" /etc/fstab; then
    echo "${WORK_DIR}/containerd-storage/io.containerd.snapshotter.v1.overlayfs /var/lib/rancher/k3s/agent/containerd/io.containerd.snapshotter.v1.overlayfs none bind 0 0" | sudo tee -a /etc/fstab
fi
if ! grep -q "containerd.content" /etc/fstab; then
    echo "${WORK_DIR}/containerd-storage/io.containerd.content.v1.content /var/lib/rancher/k3s/agent/containerd/io.containerd.content.v1.content none bind 0 0" | sudo tee -a /etc/fstab
fi

Now we can continue with the k3s setup. Let’s start by creating the directory where we must copy the k3s air-gapped container images:

sudo mkdir -p /var/lib/rancher/k3s/agent/images/

Now we can copy the k3s air-gapped container images to the directory we just created:

if [ ! -f "/var/lib/rancher/k3s/agent/images/k3s-airgap-images-amd64.tar.zst" ]; then
    sudo cp ${WORK_DIR}/k3s/k3s-airgap-images-amd64.tar.zst /var/lib/rancher/k3s/agent/images/
fi

Let’s copy the k3s binary and in to the correct place and give it execution rights:

if [ ! -f "/usr/local/bin/k3s" ]; then
    sudo cp ${WORK_DIR}/k3s/k3s /usr/local/bin/k3s
    sudo chmod +x /usr/local/bin/k3s
fi

Now we can install k3s:

if [ ! -f "/etc/systemd/system/k3s.service" ]; then
    sudo chmod +x ${WORK_DIR}/k3s/install.sh
    sudo INSTALL_K3S_SKIP_DOWNLOAD=true K3S_KUBECONFIG_MODE="644" ${WORK_DIR}/k3s/install.sh --disable traefik --disable metrics-server
    sudo chmod 600 /etc/rancher/k3s/k3s.yaml
fi

and start it if not already started:

if [ ! systemctl is-active k3s >/dev/null 2>&1 ]; then
    sudo systemctl start k3s
fi

We can now check whether the k8s cluster is up and running as expected. We will do that by waiting a few seconds and checking the number of running pods. We expect 2 running pods: coredns and local-path-provisioner:

RUNNING_PODS=$(kubectl get pods -A --no-headers 2>/dev/null | grep -c "Running" || true)
if [ "$RUNNING_PODS" -lt 2 ]; then
    echo "Waiting for k3s pods to start (sleeping 10 seconds)..."
    sleep 10
    RUNNING_PODS=$(kubectl get pods -A --no-headers 2>/dev/null | grep -c "Running" || true)
fi
if [ "$RUNNING_PODS" -lt 2 ]; then
    echo "Less than 2 pods are running. Pods status:"
    kubectl get pods -A
    echo "Exiting script..."
    exit 1
fi

If there are less than 2 running pods, something is very wrong, and we should look into it. We can exit the script at this point.

Let’s also make sure that the storage class is available, in case we need it later:

if [ -z "$(kubectl get sc 2>/dev/null | grep 'local-path')" ]; then
    echo "local-path' storage class not found"
    exit 1
fi

If the result is not what we expect we can exit the script and start looking into it.

Now we can install helm:

command -v helm >/dev/null 2>&1
if [ $? -ne 0 ]; then
    tar -zxvf ${WORK_DIR}/helm/helm-v3.15.1-linux-amd64.tar.gz -C ${WORK_DIR}/helm/
    sudo cp ${WORK_DIR}/helm/linux-amd64/helm /usr/local/bin/
    sudo chmod +x /usr/local/bin/helm
fi

To make usage easier we can configure kubeconfig for it:

if [ -f "/etc/rancher/k3s/k3s.yaml" ]; then
    mkdir -p ~/.kube
    sudo cp /etc/rancher/k3s/k3s.yaml ~/.kube/config
    sudo chown $(whoami):$(whoami) ~/.kube/config
    chmod 600 ~/.kube/config
    export KUBECONFIG=~/.kube/config
fi

Let’s also do a quick test for helm by listing all namespaces:

helm list --all-namespaces >/dev/null 2>&1
if [ $? -ne 0 ]; then
    echo "Helm is not working correctly"
    exit 1
fi

The last part is to set up MetalLB and ingress-nginx. Let’s untar the transferred file:

if [ ! -f "${WORK_DIR}/k8s-infrastructure/metallb-controller.tar" ]; then
    tar -xvf ${WORK_DIR}/temp/metallb-nginx-images.tar -C ${WORK_DIR}/k8s-infrastructure
fi

Now we can import the container images with ctr:

if ! sudo ctr -n k8s.io images ls | grep -q "quay.io/metallb/controller:v0.14.5"; then
    for f in "${WORK_DIR}/k8s-infrastructure"/*.tar; do 
        echo "Importing: $(basename "$f")"
        sudo TMPDIR=${WORK_DIR}/temp ctr -n k8s.io images import "$f"
    done
fi

Note the usage of TMPDIR when running ctr. This is important to use because the default temporary directory can also get large if you have a lot if container images to import. Also note that we did check if one image was already imported, to skip unnecessary imports. If you use other versions, you will need to update the version to check also.

Now we can apply the yaml files of MetalLB and ingress-nginx with kubectl:

kubectl apply -f "${WORK_DIR}/k8s-infrastructure/metallb.yaml" >/dev/null 2>&1
kubectl apply -f "${WORK_DIR}/k8s-infrastructure/ingress-nginx.yaml" >/dev/null 2>&1
kubectl apply -f "${WORK_DIR}/k8s-infrastructure/metallb-ip-pool.yaml" >/dev/null 2>&1

We can now run

WORK_DIR=/mnt/mydata eval "sh \$MOUNT_POINT/scripts/setupk8s.sh"

to run the script to set up k8s. Note the eval trick used to avoid duplication of the content of the WORK_DIR environment variable.

Step4

Let’s add our application. You will probably replace this step with your own custom application.

Just as we did with MetalLB and ingress-nginx, let’s import the container images:

sudo ctr -n k8s.io images ls | grep -q "docker.io/hashicorp/http-echo:latest"
if [ $? -ne 0 ]; then
    sudo TMPDIR=${WORK_DIR}/temp ctr -n k8s.io images import "${WORK_DIR}/testapp/testapp.tar"
fi

Just as before, we are using the TMPDIR environment variable and skip the import in one of the images is already present.

The last part is to apply the yaml files of our custom application with kubectl:

kubectl apply -f ${WORK_DIR}/testapp/namespace.yaml >/dev/null 2>&1
kubectl apply -f ${WORK_DIR}/testapp/deployment.yaml >/dev/null 2>&1
kubectl apply -f ${WORK_DIR}/testapp/service.yaml >/dev/null 2>&1
kubectl apply -f ${WORK_DIR}/testapp/ingress.yaml >/dev/null 2>&1

We can now run

WORK_DIR=/mnt/mydata eval "sh \$MOUNT_POINT/scripts/setupTestApp.sh"

to run the script to set up our testing application. Note the eval trick used to avoid duplication of the content of the WORK_DIR environment variable.

Final tests

From inside the virtual machine we can run

curl -H "Host: testapp.mycompany.com" http://1.2.3.4

to test if our application is reachable.

We can also navigate to http://testapp.mycompany.com in a browser on another machine to check our application. We should see our custom text Test App is Working!.

Where to go from here

If we want to configure an SSL certificate, we can do this by adding it to ingress:

apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: testapp-ingress
  namespace: testapp
  annotations:
    nginx.ingress.kubernetes.io/ssl-redirect: "true"
spec:
  ingressClassName: nginx
  rules:
  - host: testapp.mycompany.com
    http:
      paths:
      - path: /
        pathType: Prefix
        backend:
          service:
            name: testapp-service
            port:
              number: 80
  tls:
  - hosts:
    - testapp.mycompany.com
    secretName: testapp-ingress-tls-crt

Let’s not forget the secret:

apiVersion: v1
kind: Secret
metadata:
  name: testapp-ingress-tls-crt
  namespace: testapp
type: kubernetes.io/tls
data:
  tls.crt: DUMMYCRT  # base64 fullchain1.pem | tr -d '\n' on MacOS. On Linux, use base64 -w 0 fullchain1.pem
  tls.key: DUMMYKEY  # base64 privkey1.pem | tr -d '\n' on MacOS. On Linux, use base64 -w 0 privkey1.pem

Improvements can be made to the deployment of your custom application, and adding more components to the k8s infrastructure.

GitHub source