Nephio Tutorial — Setting up Nephio management and workload clusters

By Konstantinos Antonakoglou, Senior Future Networks Technologist at Digital Catapult

In this tutorial, I will describe a minimal workload deployment as an example of using Nephio (Nephio’s Release 3 specifically) on pre-provisioned Kubernetes clusters. If you don’t know what any of this means, no worries, I’ll try to explain or at least provide some source material with links. As a disclaimer, before you apply any of the commands provided make sure you are using the appropriate compute resources for testing/development purposes.

This tutorial has been tested on Digital Catapult’s private cloud infrastructure as part of the development and contributions in the R&D project REASON, short for Realising Enabling Architectures and Solutions for Open Networks. In Digital Catapult, the Future Networks team is currently using Nephio within the scope of its R&D activities, as a management and orchestration (MANO) platform for scalable service deployments. These services can be network services or other containerised applications that need to run and be managed on Kubernetes clusters. Combined with our multi-cluster Kubernetes monitoring capabilities (not described in this tutorial), it allows the intelligent orchestration of network service deployments.

Nephio is a Linux Foundation (LF) project, supported by Google, Nokia and other LF Networking members. It is “a Kubernetes-based intent-driven automation of network functions and the underlying infrastructure that supports those functions”. The project has currently a quite active community around it. You check the current development or even participate on the Nephio project’s Github page or in Slack.

As you can see in the following figure, this example deployment consists of three Kubernetes clusters. To quickly deploy the clusters, I initially tested using Minikube on 3 separate Virtual Machines (VMs) using Ubuntu 22.04 (more information later in this tutorial). Afterwards, we redeployed Nephio on Kubernetes clusters deployed on VMs with Kubespray which is using Ansible (not described in this tutorial). In summary, we deployed:

• one management Kubernetes cluster in a VM together with a Gitlab instance (and a few CLI tools)
• two workload (for example, edge or regional) Kubernetes clusters.

Figure 1 A high level diagram of the Nephio deployment of this tutorial.

A few words about Nephio

If you check the documentation pages of Nephio (actually you should go through them anyway), you will see a variety of guides/tutorials on how to install Nephio. These guides include shell scripts that automate the installation process and are also configurable through environment variables.

The Nephio documentation website provides guides for different kinds of cloud infrastructure where Kubernetes can be or is already deployed such as Kubernetes in Docker (KinD), Google Cloud Platform (GCP), Red Hat’s OpenShift and of course Bring Your Own Cluster (BYOC).

Nephio is leveraging Kubernetes for declarative management with active reconciliation and does it for 3 layers (or swim lanes as described in a few presentations):

• Cloud infrastructure
• Workload (network function) resource automation
• Workload (network function) configuration

Key components that Nephio uses and what they do

Declarative management is achieved by using Custom Resources (CRs) i.e. custom Kubernetes API extensions which can be in the form of .yaml configuration files. The CRs are given as input to custom Kubernetes operators and are stored and managed via Kubernetes.

The CR files can be bundled in the form of packages manipulated by kpt, a tool which allows the automation of editing Kubernetes configuration files. But (!) there’s more to it. These files are stored in Git or OCI repositories and managed by porch (Package Orchestration service) which is hosted in the Nephio management cluster. FYI porch is a project developed by the Nephio community, kpt is a separate project developed outside Nephio.

The Nephio operators are components/code running in Kubernetes that process the CRs, monitor their deployment and make sure that what the CRs describe is reflected by the current state of the cluster. For reconciling any changes or modifications, a key tool is ConfigSync (yet another tool by Google).

In general, using such APIs (both default and custom ones) to control the elements that comprise Nephio allows its extensibility as a micro-service based SMO. Of course, there are command line tools which we can use instead such as kubectl (Kubernetes’s CLI tool), kpt’s CLI tool and porchtl (porch’s CLI tool).

For example, the operator responsible for provisioning the cloud infrastructure (provisioning of Kubernetes clusters on a variety of cloud providers) is Cluster API. In Nephio’s case Cluster API’s operator is currently only able to provision Kubernetes on KinD.

Please note that provisioning Kubernetes clusters using Cluster API is out of the scope of this tutorial and will not be further discussed. I will also not cover workload deployments on edge clusters, as this would require its own separate follow-up tutorial. Hopefully another time though.

Bring Your Own Cluster (BYOC) — Install Prerequisites

First, we need to install Docker, kubectl and Minikube in all VMs of our set up.

Docker installation

sudo apt update 

sudo apt install apt-transport-https ca-certificates curl software-properties-common<br>curl -fsSL https://download.docker.com/linux/ubuntu/gpg | sudo gpg --dearmor -o /usr/share/keyrings/docker-archive-keyring.gpg<br>echo "deb [arch=$(dpkg --print-architecture) signed-by=/usr/share/keyrings/docker-archive-keyring.gpg] https://download.docker.com/linux/ubuntu $(lsb_release -cs) stable" | sudo tee /etc/apt/sources.list.d/docker.list > /dev/null 

sudo apt update 

apt-cache policy docker-ce 

sudo apt install docker-ce 

sudo systemctl status docker 

sudo usermod -aG docker $USER && newgrp docker 

sudo systemctl enable docker 

sudo systemctl start docker  

Single-node Kubernetes with Minikube

Even though a tidy deployment of Kubernetes clusters on a VM is preferable for scalable usage and more convenient remote access to the clusters, tools such as KinD and Minikube are useful for quick deployments of an isolated testing environment.

First let’s install kubectl (also useful for the workload clusters):

curl -LO "https://dl.k8s.io/release/$(curl -L -s https://dl.k8s.io/release/stable.txt)/bin/linux/amd64/kubectl" 

sudo install -o root -g root -m 0755 kubectl /usr/local/bin/kubectl 

Then we can install Minikube:

curl -LO https://storage.googleapis.com/minikube/releases/latest/minikube-linux-amd64 

sudo install minikube-linux-amd64 /usr/local/bin/minikube && rm minikube-linux-amd64 

minikube start 

minikube kubectl get pods 

A note on Minikube (external access with nginx)

Minikube when launched with — driver=docker (default configuration) is not reachable from outside the host. If this is something of interest to you you can try the starting it with — driver=none or install nginx and use it to deploy a reverse proxy:

sudo apt update 

sudo apt install nginx 

vi /etc/nginx/nginx.conf 

(note: I will be using vi as an editor)

then insert under the http section (under the closing brackets):

stream { 

server { 

#your host interface IP that is reachable from outside 

listen xxx.xxx.xxx.xxx:8443; 

#TCP traffic will be forwarded to the specified server 

#it's your Kubernetes IP which can be found in /.kube/configproxy_pass xxx.xxx.xxx.xxx:8443; 

} 

} 

Then restart nginx with:

sudo service nginx restart 

Install Gitlab on the Nephio Management VM

As previously mentioned, porch, a basic component in Nephio, is porch which is used to manage, manipulate and provide lifecycle capabilities for kpt packages stored in git or OCI repositories. ConfigSync deployed in each workload cluster will make sure that the workload clusters are in sync with the repositories. I will set up these repositories using Gitlab, in a single VM, in this case the same VM as the Nephio management cluster for convenience. I used this guide.

If you want you can use Docker Compose. Kubernetes deployments don’t seem to be production-ready for now.

For a manual installation:

sudo apt update 

sudo apt install ca-certificates curl openssh-server postfix tzdata perl 

cd /tmp 

curl -LO https://packages.gitlab.com/install/repositories/gitlab/gitlab-ce/script.deb.sh 

sudo bash /tmp/script.deb.sh 

sudo apt install gitlab-ce 

in /etc/gitlab/gitlab.rb also find and replace (you can use sudo vi /etc/gitlab/gitlab.rb):

external_url 'http://<MANAGEMENT_CLUSTER_HOST_IP>:9876' 

puma['port'] = 8085 

You can now start creating repositories (or “new projects” in gitlab lingo).

Nephio repositories

In Nephio there are two main kinds of repositories, deployment repositories and blueprint repositories. I think it’s easier to explain what a blueprint repository is by first explaining what a deployment repository is.

The deployment repositories are in sync with the workload clusters and include kpt package revisions that have been customized/manipulated to include all necessary configuration data (e.g. IPs related to the specific workload cluster interfaces or equipment) and have been approved for deployment (either manually or automatically). These are the hydrated versions of blueprint repositories.

The blueprint repositories are the dry versions that do not include custom changes or configuration data related to a specific kind of deployment.

You can go on and create the following repositories:

1. A local blueprint repository the Nephio management cluster will use
2. A deployment repository for Workload cluster 1 (or edge cluster)
3. A deployment repository for Workload cluster 2 (or regional cluster)

Nephio Management Cluster Installation

It’s time to install two important CLI tools. These are the tools that control some of the software we are interested in. I will explain later what they actually do.

Install kpt CLI tool

Full installation details for kpt are here.

For installing version v1.0.0-beta.44, download the binary, move it to /usr/local/bin (and rename it to kpt at the same time using mv) and update the execution permissions:

wget https://github.com/GoogleContainerTools/kpt/releases/download/v1.0.0-beta.44/kpt_linux_amd64 

mv ./kpt_linux_amd64 /usr/local/bin/kpt 

chmod +x /usr/local/bin/kpt 

Install porchctl

When Porch was ported to Nephio, the kpt alpha rpkg commands in kpt were moved into a new command called porchctl.

mkdir porch-install 

wget https://github.com/nephio-project/porch/releases/download/v3.0.0/porchctl_3.0.0_linux_amd64.tar.gz 

tar -xvf porchctl_3.0.0_linux_amd64.tar.gz -C porch-install 

sudo mv ./porch-install/porchctl /usr/local/bin/ 

Install the Nephio Base Components & Operators

Now it’s time to install the Nephio base components and operators in the management Kubernetes cluster as described here and here. Perhaps some of these operators can be considered optional, so feel free to mix and match the ones you need.

To keep things tidy, create a nephio-install folder, cd in it and continue with the component installation commands.

The nephio-install folder will be used by kpt to download the component repos:

mkdir nephio-install 

cd nephio-install 

Network Config Operator

This is a controller for applying configuration to routers and switches. Installation instructions are from here.

Fetch the package from the Nephio git repository using kpt , run the kpt functions, and then apply the package:

kpt pkg get --for-deployment https://github.com/nephio-project/catalog.git/nephio/optional/network-config@@origin/v3.0.0 

kpt fn render network-config

kpt live init network-config 

kpt live apply network-config --reconcile-timeout=15m --output=table 

The first command will simply create a network-config folder inside the nephio-install folder we previously created.

We can check if the network-config operator is installed with kubectl. This command is useful for checking all other deployments:

kubectl get pods -A 

Resource Backend

The resource backend operator provides IP and VLAN allocation.

Same as before we use kpt:

kpt pkg get --for-deployment https://github.com/nephio-project/catalog.git/nephio/optional/resource-backend@@origin/v3.0.0 

kpt fn render resource-backend 

kpt live init resource-backend 

kpt live apply resource-backend --reconcile-timeout=15m --output=table 

Porch Operator

The porch operator provides the custom Kubernetes APIs for Repositories (package repositories), PackageRevisions, PackageRevisionResources, PackageVariants, and PackageVariantSets. Nephio relies on it for having a package inventory and for being able to clone, and mutate packages. It also provides the API layer that shields the Nephio components from direct interaction with the Git (or OCI) storage layer.

Same as before we use kpt:

kpt pkg get --for-deployment https://github.com/nephio-project/catalog/nephio/core/porch@origin/v3.0.0 

kpt fn render porch

kpt live init porch 

kpt live apply porch --reconcile-timeout=15m --output=table 

Nephio Operator

The nephio-operator kpt package includes a bundle of Kubernetes CRs and essentially deploys two pods the nephio-controller and the token-controller in the nephio-system namespace.

We will break the kpt installation in two parts because we need to edit a file before applying the package to point to our Gitlab instance. So first we fetch the package:

kpt pkg get --for-deployment https://github.com/nephio-project/catalog.git/nephio/core/nephio-operator@origin/v3.0.0 

Note: Before installing the Nephio Operators edit the following yaml files to use Gitlab (which in our case is in http://127.0.0.1:9876):

vi nephio-operator/app/controller/deployment-token-controller.yaml

- name: GIT_URL 
  value: http://127.0.0.1:9876 

and then we execute the kpt commands:

kpt fn render nephio-operator 

kpt live init nephio-operator 

kpt live apply nephio-operator --reconcile-timeout=15m --output=table

If you want to authenticate access to your Gitlab repositories and therefore don’t use public repositories, for the nephio-operator to be able to access your private git repos you need to store a secret in Kubernetes which will include the username and password of the Gitlab account that has access to the repositories:

kubectl apply -f  - <<EOF 
apiVersion: v1 
kind: Secret 
metadata: 
name: git-user-secret 
namespace: nephio-system 
type: kubernetes.io/basic-auth 
stringData: 
username: <GITLAB_USERNAME> 
password: <GITLAB_PASSWORD> 
EOF 

Nephio Web UI

The final (optional) component of the Nephio Management cluster is the Nephio Web UI. It’s convenient to use it but all actions can be achieved via command line (or code) as well.

kpt pkg get --for-deployment https://github.com/nephio-project/catalog.git/nephio/optional/webui@origin/v3.0.0 

kpt fn render webui 

kpt live init webui 

kpt live apply webui --reconcile-timeout=15m --output=table --inventory-policy=adopt 

Please notice there’s a difference between the URL here (the correct one currently) and the URL in the Nephio documentation (which doesn’t work)

For external access to the Web UI via a Kubernetes Ingress/Load Balancer or Gateway with or without authentication you need to dig deeper into the previous documentation link and perhaps other Kubernetes tutorials.

In our case, we can use SSH tunneling on a local terminal first:

ssh <USERNAME>@<VM_IP> -L 7007:localhost:7007 

and then port-forwarding on Kubernetes:

kubectl port-forward --namespace=nephio-webui svc/nephio-webui 7007 

Now the Web UI should be accesssible from your browser on http://localhost:7007/config-as-data.

Register repositories

On the management cluster, you can register repositories via command line or by applying CRs on Kubernetes.

For deployment repositories:

kpt alpha repo register \ 
  --namespace default \ 
  --repo-basic-username=<add_a_username>\ 
  --repo-basic-password=<add_a_password>\ 
  --create-branch=true \ 
  --deployment=true \ 
  http://<GITLAB_IP>:9876/<USER>/<DEPLOYMENT_REPOSITORY_NAME>.git 

For blueprint repositories you need to change the — deployment value to false.

The CR way of doing it is:

cat << EOF | kubectl apply -f - 
apiVersion: v1 
kind: Secret 
metadata: 
 name: organizational-blueprints 
type: kubernetes.io/basic-auth 
stringData: 
  username: <GITLAB_USERNAME> 
  password: <GITLAB_PASSWORD> 
— 
apiVersion: config.porch.kpt.dev/v1alpha1 
kind: Repository 
metadata: 
  name: <YOUR_BLUEPRINT_REPO_NAME> 
  namespace: default 
  labels: 
    kpt.dev/repository-access: write 
    kpt.dev/repository-content: organizational-blueprints 
spec: 
  content: Package 
  deployment: false 
  git: 
branch: main 
directory: / 
repo: http://<GITLAB_IP>:9876/<USER>/<BLUEPRINT_REPO_NAME>.git 
secretRef: 
    name: organizational-blueprints 
  type: git 
EOF 

You can choose the Nephio repository category (as in the categories shown in the Web UI’s front page) by changing the kpt.dev/repository-content value. For example, you can change it from organizational-blueprints to external-blueprints.

Nephio Workload Cluster installation

In each workload cluster we first need to install kpt cli tool (the same process as before). This will allow us to install ConfigSync and RootSync.

Install ConfigSync

kpt pkg get --for-deployment https://github.com/nephio-project/catalog.git/nephio/core/configsync@@origin/v3.0.0 

kpt fn render configsync 

kpt live init configsync 

kpt live apply configsync --reconcile-timeout=15m --output=table 

Install RootSync

First, we fetch the RootSync package:

kpt pkg get https://github.com/nephio-project/catalog.git/nephio/optional/rootsync@@origin/v3.0.0 

and then we edit /rootsync/rootsync.yaml:

spec: 
   sourceFormat: unstructured 
   git: 
repo: <HTTP URL OF YOUR WORKLOAD REPOSITORY> 
  branch: main 
  auth: none 
  period: 15s 

If credentials are needed to access your repository then copy the token name from previous section and provide it in ./rootsync/rootsync.yaml

spec: 
  sourceFormat: unstructured 
  git: 
repo: <HTTP URL OF YOUR WORKLOAD REPOSITORY> 
branch: main 
auth: token 
secretRef: 
   name: <TOKEN-NAME> 

Then execute the kpt commands:

kpt live init rootsync 

kpt live apply rootsync --reconcile-timeout=15m --output=table 

Add Workload CRDs in Edge workload cluster

Workload CRDs are required to manage network functions.

kpt pkg get --for-deployment https://github.com/nephio-project/catalog.git/nephio/core/workload-crds@@origin/v3.0.0 

kpt live init workload-crds 

kpt live apply workload-crds --reconcile-timeout=15m --output=table 

Other workload cluster controllers

You can deploy other workload cluster controllers like Multus for managing network interfaces of Pods or FluxCD for making deployments by syncing Helm repositories with your cluster. For the latter there is actually a tutorial provided by the Nephio community that will also introduce you to the Web UI.