Cloud-Native DevLab — Rebuild

About 5 years ago I created a blog post (with an accompanying GitHub repo) on how to build an on-premises devlab that was about as cloud native as I could make it at the time. I had a ton of views and reads, so it must have struck a chord with people. (No idea if people built similar labs, but the need for a lab seems to have been a thing.) I based myself on Azure Stack HCI and got it working both on a single HPE Proliant Microserver and a two-node redundant configuration. This was of course way before the RAMpocalypse so no selling of organs was necessary to achieve this 🙂

Small Form Factor for your Kubernetes lab

Today, I’m having another go with “Azure Local Small Form Factor” following the announcements at Build 2026. This is based on Azure Linux and a process for bootstrapping into Azure Arc.

[embed]Introducing small form factor infrastructure: embed intelligence into physical systems Today at Microsoft Build 2026, we're announcing new capabilities to help organizations everywhere get started extending…techcommunity.microsoft.com

But before diving into the new things, let’s do a quick recap of the road in between (rant warning).

When I created the lab originally it had some requirements around networking and storage to figure out, and after installing the Azure Stack HCI operating system you ran a bunch of PowerShell to get things up and running before doing a small connection step at the end to make it sync up with Azure. Then you could install Azure Kubernetes Service (AKS) on-premises and even deploy resources with Bicep through the ARM control plane. After a couple of version upgrades Microsoft decided to turn this around by moving “all the work” to Azure; just a minor job installing the OS locally and then pushing config from the cloud to on-prem. This broke the guide of course, but it made things a lot harder if you didn’t use fully supported hardware so I more or less abandoned that approach. Luckily, I found another way to do things. Installing plain Windows Server and running AKS on top of that instead since that was a supported scenario. I even installed Azure Container Apps onto this rig. (It was deployed as an extension to an AKS cluster.) This has been working fairly well if you ignore the joke that is Windows Admin Center (which is the intended management UI). So, Microsoft has of course decided to deprecate running AKS on Windows Server. (I can see some business reasons why you don’t want to support too many scenarios, but irritating nonetheless.)

In parallel they have offered AKS Edge Essentials which lets you run a small-scale cluster (K8s or K3s) on Windows. (Intended for Windows 10/11 but I ran it on Windows Server.) And they have rebranded Azure Stack to Azure Local. You can run Azure Local on top of Hyper-V with nested virtualization for a lab environment instead of certified hardware, but it’s still a complex solution. They soft announced the low-cost hardware option last year, but that incarnation apparently never came out of private preview. Which leads us to today where they have an Azure Local solution using Azure Linux marketed towards low-cost/small form factor hardware.

If you followed along through the rambling, you’re probably thinking I’m an idiot for not nuking Windows on these boxes a long time ago and installing Linux already since Kubernetes is the de facto standard these days and doesn’t require guidance from Microsoft. I’m not saying I’ve just been waiting patiently for Microsoft to act here. I have tested Kubernetes on Linux VMs, and I even have K3s running on my Asus GX10 (an Nvidia DGX Spark clone) for cranking out in-house AI-slop. That one is connected through Azure Arc as well and is able to leverage a managed identity to reach into Azure resources. (Not sure if that warrants a walkthrough of its own.) Still want to keep up to date with what a fully Azure-integrated experience can provide so that led me to deciding it was worth another shot.

Pausing for a moment on our options here I have to concede the cloud itself is also an option. With AKS (in Azure) you get a baseline cost which isn’t homelab friendly, but with a service like Azure Container Apps you can truly scale to zero cost-wise. Actually spinning up the containers have a cost, but unless you have a hot instance running at all times it doesn’t have to be that bad. Depending on your needs and exerting cost control you can do a lot without bothering with your own hardware. Not to mention, you can get quite far with the laptops of today as well. But for me the whole point of this exercise is testing dedicated hardware 🙂

Microsoft lists a couple of devices they consider validated, so I went with an ASUS NUC 15 Pro to play it safe. Virtual machines are also said to work so feel free to go down the virtualized path if you like.

All of this is in a preview state at the moment so appropriate warnings apply. The good thing is that there is no cost while in preview. I would expect this to change at a later point in time, but I don’t know if they will go with “hobbyist” pricing or something more enterprise-like.

Bootstrapping the NUC The first phase is installing an operating system on the device. You do this by formatting a USB stick that will provision an installer, which in turn will install a thin management layer. Then you connect it to Azure to download the actual operating system and run through the installation remotely. Just follow the official guide here:

https://learn.microsoft.com/en-us/azure/azure-local/small-form-factor/small-form-factor-installation?view=azloc-2605&tabs=usb-drive

I found it to be slightly random as to whether it stored the .pem file you need on the device or the USB stick, and it also seemed to get stuck sometimes on the “finalizing security” step where I just ended up rebooting it for it to change to “succeeded”. Still, fairly quick and easy. Outside our lab usage one of the main points of this offering is to make it easy to deploy edge devices, so this all makes sense for enabling the local IT guy to do the basics on his own before passing off control to HQ.

Once you have the voucher it’s on to installing the OS which is initiated from the Azure portal:

https://learn.microsoft.com/en-us/azure/azure-local/small-form-factor/small-form-factor-connect-portal?view=azloc-2605

Seems to take about 25 minutes like the docs say, and I had no issues here.

Deploying a container host solution This box is intended for running containers, which fits nicely into a developer lab these days. You have three options to choose from here: Docker, K3s and AKS.

I like using Docker on the device where I code, and it also works good for things like hosting a couple of solutions through a docker-compose file, but it doesn’t really allow for “complete flows”.

K3s is a good way to have Kubernetes running in no time with sensible defaults and for many this might be the sweet spot. Should have less preview restrictions than AKS at the moment as K3s is a mature solution.

AKS is “proper” Kubernetes. The interesting thing is that this is another preview service called “AKS on bare metal” where you don’t go through the overlay of running Kubernetes on virtual machines (which is what you do when running AKS on Azure Stack HCI or Windows Server). I like bleeding edge so I went with this option for now.

There are official docs for this too:

https://learn.microsoft.com/en-us/azure/aks/aksarc/aks-bare-metal-system-requirements

I had issues getting all API-versions registered for the Microsoft.HybridContainerService in my intended subscription, so I had to use a different sub. No idea why, but just a heads-up that could be a thing for you as well.

I went for full integration with Entra and I didn’t have to create any groups. That means you have to assign yourself the “Azure Arc Kubernetes Cluster Admin” role on the cluster resource afterwards to be able to manage the cluster.

To connect to the cluster and do a quick status you fire up one terminal where you proxy the connection:

az connectedk8s proxy — name <cluster-name> — resource-group <resource-group>

Followed by a plain old kubectl in a different terminal tab/windows:

kubectl get nodes

Viewing the Kubernetes resources from the Azure portal has also been wired up automatically, and while it does not show everything I want it to show it’s an easy way to check things are mostly good.

If you have a preferred set of yamls to get going with K8s you should be ready to go by now. I however, want to go through a couple more things.

Workload Identity Federation One of the advantages of running Kubernetes on Azure is Workload Identity for enabling authentication without using client secrets. (Advantage in the context of using Entra and having other resources deployed to Azure.) This also extends to Arc-based resources. You would typically enable it at connect time with az connectedk8s connect, but you can also enable it afterwards with az connectedk8s update — name “clusterName” — resource-group “rg” — enable-oidc-issuer — enable-workload-identity. No luck with that here since you get the following error:

Updating a Provisioned Cluster is not supported from the Connected Cluster CLI. Please use the ‘az aksarc update’ CLI command.

Which is sort of right, except for the fact that the — enable-oidc-issuer — enable-workload-identity parameters only apply on “create” and not “update”. We don’t have direct control of this at cluster creation time when using the Azure portal since it is not exposed in the creation wizard. It’s not included in the Bicep template provided either for that matter, but since the connectedCluster resource type supports it, I altered the following Bicep to make it work:

resource connectedCluster 'Microsoft.Kubernetes/connectedClusters@2026-05-01' = {
  name: clusterName
  location: location
  identity: {
    type: 'SystemAssigned'
  }
  kind: 'ProvisionedCluster'
  properties: {
    agentPublicKeyCertificate: ''
    aadProfile: {
      enableAzureRBAC: enableAzureRbac
      adminGroupObjectIDs: []
    }
    oidcIssuerProfile: {
      enabled: true
    }
    securityProfile: {
      workloadIdentity: {
        enabled: true
      }
    }
  }
  tags: mergedTags
  dependsOn: [
    logicalNetwork
  ]
}

The rest of the Bicep can be found here (you might want to bump the API version dates for the other resources while you’re at it):

https://github.com/Azure/aksArc/blob/main/deploymentTemplates/aks-baremetal-bicep/cluster-create.bicep

Creating Workload Identity Enabling support doesn’t automatically configure it all, so there’s a couple of extra steps required for that.

Create a user-assigned managed identity:

# Get OIDC Issuer Url
export OIDC_ISSUER="$(az connectedk8s show --name "${CLUSTER_NAME}" --resource-group "${RESOURCE_GROUP}" --query "oidcIssuerProfile.issuerUrl" --output tsv)"

# Create UAMI
az identity create --name "${USER_ASSIGNED_IDENTITY_NAME}" --resource-group "${RESOURCE_GROUP}" --location "${LOCATION}" --subscription "${SUBSCRIPTION}"

# GET UAMI ID
export USER_ASSIGNED_CLIENT_ID="$(az identity show --resource-group "${RESOURCE_GROUP}" --name "${USER_ASSIGNED_IDENTITY_NAME}" --query 'clientId' --output tsv)"

Create Kubernetes Service Account by applying the following yaml (after you’ve fixed the placeholders):

# SA
apiVersion: v1
kind: Namespace
metadata:
  name: "${SERVICE_ACCOUNT_NAMESPACE}"
---
apiVersion: v1 
kind: ServiceAccount 
metadata: 
  annotations: 
    azure.workload.identity/client-id: "${USER_ASSIGNED_CLIENT_ID}" 
  name: "${SERVICE_ACCOUNT_NAME}" 
  namespace: "${SERVICE_ACCOUNT_NAMESPACE}"

Create a Federated Identity Credential:

az identity federated-credential create --name ${FEDERATED_IDENTITY_CREDENTIAL_NAME} --identity-name "${USER_ASSIGNED_IDENTITY_NAME}" --resource-group "${RESOURCE_GROUP}" --issuer "${OIDC_ISSUER}" --subject system:serviceaccount:"${SERVICE_ACCOUNT_NAMESPACE}":"${SERVICE_ACCOUNT_NAME}" --audience api://AzureADTokenExchange

More details can be found here:

https://learn.microsoft.com/en-us/azure/azure-arc/kubernetes/workload-identity

Certificates with Cert-manager

With the identities in place, I usually recommend installing cert-manager. You can install cert-manager in a couple of ways, but Microsoft has a preview out for a managed version of this which should mean it’s even simpler to maintain:

https://learn.microsoft.com/en-us/azure/azure-arc/kubernetes/cert-manager-overview

You basically just run a single line to install it as an extension:

az k8s-extension create — resource-group ${RESOURCE_GROUP} — cluster-name ${CLUSTER_NAME} — cluster-type connectedClusters — name “azure-cert-management” — extension-type “microsoft.certmanagement” — config cert-manager.config.enableGatewayAPI=true

You still have to enroll certs through creating DNS records so it requires assigning permissions as well as annotating the managed identity though.

Which leads me to a restriction in the current preview.

Exposing Services

It seems the Gateway API CRDs are already installed:

kubectl get crds | grep “gateway.networking.k8s.io” kubectl get crd gateways.gateway.networking.k8s.io -ojsonpath={.metadata.annotations} | jq

{
  "api-approved.kubernetes.io": "https://github.com/kubernetes-sigs/gateway-api/pull/3328",
  "gateway.networking.k8s.io/bundle-version":"v1.2.0",
  "gateway.networking.k8s.io/channel":"standard"
}

Apparently, it’s an older version (1.2.0) whereas K8s 1.34.3 should support 1.3.0 on AKS. So, here’s me thinking let’s just deploy Traefik or something. Right…

Unfortunately, this is what the current docs say for deploying an application “Use NodePort for external access: LoadBalancer type isn’t supported during preview”. It prevented me from getting Traefik deployed as well. The docs show how to deploy an nginx image and expose through NodePort so it works as described. (I wonder if they see the irony of using nginx as the sample app.)

I would expect this to be fixed eventually, but probably a matter of prioritization. The video they published for demoing what you can do with Azure Local low-cost hardware showed a robot arm reacting to voice and moving some objects around on a table. Nothing wrong with that, but it didn’t have a web interface and probably consisted of agents running on the device. Crossing our fingers it will come later (because proper K8s needs proper ingressing). Traefik was just what I tested as a fairly standard solution, but I’m ok with something more Azure-like as long as I’m not required to use one of the pricey AppGW/Front Door SKUs.

GitOps I’m not religious when it comes to practicing GitOps, and I have some issues with the term as well. No, you should not deploy to production based on kubectl from your laptop, but modern CI/CD practices handle this already through Git repos. Push-based deployments work nicely for things like Azure Container Apps. And pull-based deployments are a better fit for other use cases.

Anyways, that’s a digression. I don’t mind enabling the option 🙂 AKS bare metal supports enabling Flux and ArgoCD through the Azure portal (like the other AKS versions). Guessing ArgoCD might conflict with the NodePort requirement by default for the web ui. The Azure pricing calculator lists GitOps as being free for most on-premises variations of Kubernetes, but bare metal is not listed so not sure if it is covered yet or not.

Personally, I’m interested in testing out Flux Operator and Gitless GitOps: https://fluxoperator.dev

I went with Helm for the installation:

helm install flux-operator oci://ghcr.io/controlplaneio-fluxcd/charts/flux-operator -namespace flux-system

And to just get the defaults up and running you can use the supplied example:

apiVersion: fluxcd.controlplane.io/v1
kind: FluxInstance
metadata:
  name: flux
  namespace: flux-system
  annotations:
    fluxcd.controlplane.io/reconcileEvery: "1h"
    fluxcd.controlplane.io/reconcileArtifactEvery: "10m"
    fluxcd.controlplane.io/reconcileTimeout: "5m"
spec:
  distribution:
    version: "2.x"
    registry: "ghcr.io/fluxcd"
    artifact: "oci://ghcr.io/controlplaneio-fluxcd/flux-operator-manifests"
  components:
    - source-controller
    - kustomize-controller
    - helm-controller
    - notification-controller
    - image-reflector-controller
    - image-automation-controller
  cluster:
    type: kubernetes
    size: medium
    multitenant: false
    networkPolicy: true
    domain: "cluster.local"
  kustomize:
    patches:
      - target:
          kind: Deployment
        patch: |
          - op: replace
            path: /spec/template/spec/nodeSelector
            value:
              kubernetes.io/os: linux
          - op: add
            path: /spec/template/spec/tolerations
            value:
              - key: "CriticalAddonsOnly"
                operator: "Exists"

You could add annotations for the workload identity here, but we’re not covering that part of the loop at the moment.

The web ui (which I sometimes wish for in plain Flux) will require the NodePort config, or a port-forwarding:

kubectl -n flux-system port-forward svc/flux-operator 9080:9080

Next steps Alright, we have an AKS cluster — now what? Well, we got things up and running fairly easily compared to earlier offerings. I’m looking forward to see what upgrades will come; for instance if Azure Linux 4 brings anything new to the table. (Currently it’s running Azure Linux 3.)

Foundry Local is in a private preview that it’s possible to sign up for. I like the concept as it could potentially be a user-friendly way of setting up Model-as-a-Service (MaaS). It currently seems to focus on smaller models, but in this NUC I don’t have any other GPU than the onboard Intel Arc. (And the NPU doesn’t do much yet.)

Running agents is more about the CPU though (I can do the inference on my GX10), and this is probably what I will be exploring next (and sharing if I think it may benefit others). Of interest here is that since we’ve onboarded to Azure Arc and have Workload Identity it should be possible to use the Microsoft Entra ID Auth SDK (sidecar):

https://learn.microsoft.com/en-us/entra/msidweb/agent-id-sdk/installation

I’m open to suggestions if there’s anything obvious I’ve missed here of course 🙂