How to Build a Kubernetes Operator from Scratch — The Guide I Wish Existed When I Started

I tried to learn Kubernetes Operators three times. Each time, I gave up within an hour.

The official docs assumed I already understood everything. The tutorials were either too simple — “here’s a hello-world operator that does nothing useful” — or too complex — “here’s a production-grade operator with webhooks, finalizers, and 400 lines of reconciliation logic.” Nothing existed in between.

Not a Member?? Click Here

So I wrote the guide that would have saved me those three failed attempts. We’re building a real operator that does something tangible: you apply a single WebApp YAML, and the operator automatically creates a Deployment, a Service, and wires them together. One resource in, three resources out. Forty-five minutes of actual work.

Before You Start

• Go 1.21+ installed (go version to check)
• Docker installed and running
• A Kubernetes cluster — Minikube works perfectly for this
• kubectl configured and pointing at your cluster
• kubebuilder CLI installed (curl -L -o kubebuilder https://go.kubebuilder.io/dl/latest/$(go env GOOS)/$(go env GOARCH))
• Basic Go knowledge — you need to understand structs, functions, and error handling

Why I Failed 3 Times (And What Was Missing)

Every explanation I found started with jargon. “Operators extend the Kubernetes API using Custom Resource Definitions and implement a control loop via a controller-runtime reconciler.” Sure. That tells me nothing about what I’m actually building.

Here’s what an operator actually is, in three sentences:

A Kubernetes Operator is custom code that watches for changes to a Custom Resource — one that you define — and takes action. Think of it as teaching Kubernetes a new skill. You define a “WebApp” resource, and the operator automatically creates the Deployment, Service, and anything else it needs to run.

That’s it. You invent a new Kubernetes object. You write the code that responds when someone creates, updates, or deletes that object. Kubernetes handles the rest — the API server, etcd storage, event watching — for free.

[embed]Best Practices for Kubernetes Security When Your Cluster Becomes the Enemy: A Real-World Kubernetes Security Wake-Up Calldevsecopsai.today

Teaching Kubernetes a New Trick

By the end of this guide, you’ll apply this YAML to your cluster:

apiVersion: apps.example.com/v1
kind: WebApp
metadata:
  name: my-app
spec:
  image: nginx:latest
  replicas: 3
  port: 80

And your operator will automatically create a three-replica Deployment running nginx, a ClusterIP Service exposing port 80, and update the WebApp’s status with the number of available replicas. Delete the WebApp, and everything cleans up. No manual intervention. No scripts. If you’re wondering how Docker and Kubernetes fit together at this level, that piece covers the foundational relationship.

What We’re Building

1. Scaffold the Project in Two Commands
2. Define What a “WebApp” Actually Is
3. The Reconcile Function: Where the Magic Happens
4. Apply One YAML, Get Three Resources. Automatically.
5. Ship It — Build and Deploy the Operator to the Cluster

Scaffold the Project in Two Commands

Kubebuilder generates the entire project structure for you — the API types, the controller skeleton, the Makefile, the Dockerfile, the RBAC manifests. You don’t write boilerplate. You fill in the interesting parts.

# Initialize the project
kubebuilder init \
  --domain example.com \
  --repo github.com/youruser/webapp-operator
# Create the API and controller
kubebuilder create api \
  --group apps \
  --version v1 \
  --kind WebApp \
  --resource --controller

That second command is where kubebuilder asks: “Create Resource? [y/n]” and “Create Controller? [y/n].” Say yes to both. It generates two files that matter: api/v1/webapp_types.go (your CRD definition) and internal/controller/webapp_controller.go (your reconciliation logic).

Checkpoint: Your project directory has an api/ folder, an internal/controller/ folder, a Makefile, and a Dockerfile. Run go build ./... — it should compile without errors.

Gotcha: The --repo flag must match your Go module path exactly. If it doesn't match, imports break everywhere and the errors are cryptic. Get this right the first time.

Define What a “WebApp” Actually Is

Open api/v1/webapp_types.go. This file defines what fields your WebApp resource has — its spec (what the user wants) and its status (what's actually happening). Kubebuilder generated placeholder structs. Replace them with something meaningful:

// WebAppSpec defines the desired state
type WebAppSpec struct {
    Image    string `json:"image"`
    Replicas int32  `json:"replicas"`
    Port     int32  `json:"port"`
}
// WebAppStatus defines the observed state
type WebAppStatus struct {
    AvailableReplicas int32  `json:"availableReplicas"`
    URL               string `json:"url,omitempty"`
}

Three spec fields. That’s all the user needs to provide: which image to run, how many replicas, and which port to expose. The status fields let the operator report back — how many replicas are actually ready and where to reach the service.

Now regenerate the CRD manifests and install them on your cluster:

make manifests        # Generates CRD YAML from your Go types
make install          # Applies the CRD to your cluster

Checkpoint: Run kubectl get crd — you should see webapps.apps.example.com in the list. Kubernetes now knows what a WebApp is. It just doesn't know what to do with one yet.

Gotcha: Every time you change webapp_types.go, you must run make manifests again. The CRD YAML is generated from Go struct tags. If you skip this step, the cluster has stale field definitions and your YAML will silently drop unknown fields.

The Reconcile Function: Where the Magic Happens

This is the heart of every operator. The Reconcile function runs every time something happens to a WebApp resource — created, updated, deleted. Its job is simple: look at what the user wants (the spec), look at what exists (the cluster state), and make them match.

Open internal/controller/webapp_controller.go. The scaffolded Reconcile function is empty. Here's the logic we need to add. First, fetch the WebApp resource:

func (r *WebAppReconciler) Reconcile(ctx context.Context, req ctrl.Request) (ctrl.Result, error) {
    log := log.FromContext(ctx)
// Fetch the WebApp instance
    webapp := &appsv1.WebApp{}
    if err := r.Get(ctx, req.NamespacedName, webapp); err != nil {
        return ctrl.Result{}, client.IgnoreNotFound(err)
    }
    log.Info("Reconciling WebApp", "name", webapp.Name)

Next, create or update the Deployment. This is the pattern you’ll use for every child resource — check if it exists, create it if it doesn’t, update it if the spec has changed:

// Define the desired Deployment
    deploy := &appsv1.Deployment{ObjectMeta: metav1.ObjectMeta{
        Name: webapp.Name, Namespace: webapp.Namespace,
    }}
_, err := ctrl.CreateOrUpdate(ctx, r.Client, deploy, func() error {
        replicas := webapp.Spec.Replicas
        deploy.Spec.Replicas = &replicas
        deploy.Spec.Selector = &metav1.LabelSelector{
            MatchLabels: map[string]string{"app": webapp.Name},
        }
        deploy.Spec.Template = corev1.PodTemplateSpec{
            ObjectMeta: metav1.ObjectMeta{Labels: map[string]string{"app": webapp.Name}},
            Spec: corev1.PodSpec{Containers: []corev1.Container{{
                Name: "webapp", Image: webapp.Spec.Image,
                Ports: []corev1.ContainerPort{{ContainerPort: webapp.Spec.Port}},
            }}},
        }
        return controllerutil.SetControllerReference(webapp, deploy, r.Scheme)
    })

That last line — SetControllerReference — is critical. It tells Kubernetes that this Deployment is owned by the WebApp resource. When the WebApp is deleted, Kubernetes garbage-collects the Deployment automatically. Without it, you get orphaned resources littering your cluster.

Add the same pattern for a Service, then update the WebApp’s status:

// Update status with available replicas
    webapp.Status.AvailableReplicas = deploy.Status.AvailableReplicas
    webapp.Status.URL = fmt.Sprintf("http://%s:%d", webapp.Name, webapp.Spec.Port)
    r.Status().Update(ctx, webapp)
return ctrl.Result{}, err
}

Run the operator locally against your cluster to test:

make run

Checkpoint: The terminal shows “Starting Controller Manager” with no errors. The operator is running locally, connected to your cluster, waiting for WebApp resources to appear.

Gotcha: If you forget SetControllerReference, deleting a WebApp leaves its Deployment and Service behind. You'll end up with ghost resources that no one owns. Always set the owner reference.

[embed]Best Practices for Policy-as-Code Automation: I Blocked Production for 3 Hours and Lost $50K… The Deployment That Never Happeneddevsecopsai.today

Apply One YAML, Get Three Resources. Automatically.

With the operator running in your other terminal, apply the WebApp resource:

# Save this as webapp-sample.yaml
apiVersion: apps.example.com/v1
kind: WebApp
metadata:
  name: my-app
spec:
  image: nginx:latest
  replicas: 3
  port: 80

kubectl apply -f webapp-sample.yaml

Now watch the operator’s terminal. You’ll see the reconcile log fire. Then check your cluster:

kubectl get webapp              # Your custom resource
kubectl get deployment          # Auto-created by operator
kubectl get svc                 # Auto-created by operator
kubectl get pods                # 3 replicas running

One YAML applied. Three resources created. Zero manual work. This is what operators were designed for — encoding operational knowledge into software. The same pattern scales to managing databases, message queues, certificates, anything. For teams running microservices at scale, operators turn repetitive manual tasks into self-managing infrastructure.

Now test the update path. Change the replicas from 3 to 5 in your YAML and reapply. The operator reconciles and scales the Deployment. Change the image to httpd:latest and reapply. The operator updates the Deployment spec and Kubernetes rolls out the new image.

Finally, test cleanup:

kubectl delete webapp my-app
kubectl get deployment,svc,pods  # Everything gone. Clean.

Checkpoint: After deletion, kubectl get deployment shows no Deployment for my-app. The Service is gone. All pods are terminated. Owner references handled the cleanup automatically.

Ship It — Build and Deploy the Operator to the Cluster

Running make run locally is great for development. For production, the operator needs to live inside the cluster as a pod, running continuously, surviving restarts, monitored like any other workload.

Kubebuilder’s Makefile handles the entire packaging process:

# Build the operator image and push to your registry
make docker-build docker-push IMG=your-registry/webapp-operator:v1
# Deploy the operator into the cluster
make deploy IMG=your-registry/webapp-operator:v1

This creates a namespace (webapp-operator-system), deploys the operator as a Deployment with one replica, installs the CRDs, and applies RBAC rules that grant the operator permission to manage Deployments, Services, and WebApp resources.

# Verify the operator is running inside the cluster
kubectl get pods -n webapp-operator-system

Now remove your local make run process. The in-cluster operator handles everything from here. Apply a WebApp from anywhere — it just works. Make sure your cluster's security practices are solid before giving any operator cluster-level permissions.

Checkpoint: The operator pod shows Running status. Apply a WebApp resource — the Deployment and Service appear within seconds. Delete it — everything cleans up. The operator works identically to your local run, but now it's a self-contained cluster citizen.

Gotcha: RBAC will bite you here. The operator needs explicit permissions to create Deployments and Services. Kubebuilder generates these from //+kubebuilder:rbac comments in your controller file. If you added new resource types but forgot the RBAC annotations, the operator logs will show "forbidden" errors. Add the annotations and run make manifests again.

[embed]I Tried Passing the AWS Solutions Architect Exam Using Only AI — Here’s What Happened After 60 Days AI helped me move faster — until it confidently taught me the wrong AWS. Here’s the verification loop that kept my…abdoboshy.medium.com

Where to Go From Here

You have a working operator. Here’s what turns it from a demo into production infrastructure:

• Validation webhooks — reject invalid specs (negative replicas, empty image names) before they hit the controller
• More child resources — add Ingress creation, ConfigMap management, HPA auto-scaling rules
• Integration tests — kubebuilder includes an envtest framework that spins up a real API server for testing without a full cluster
• Helm packaging — distribute your operator as a Helm chart so teams can install it with helm install
• AI-assisted development — I’ve been experimenting with using ChatGPT to accelerate K8s automation, and operator development is a natural fit for AI-assisted coding

Three failed attempts. Three different tutorials that left me stranded. So I wrote the guide I needed: define your resource, write the reconcile loop, and let Kubernetes do the rest. Forty-five minutes of work. One operator that manages itself forever.

Operators aren’t magic. They’re a pattern — watch, compare, act — wrapped in a framework that handles 90% of the plumbing for you. The hard part was never the code. It was finding an explanation that didn’t assume you already knew everything.

Now you have that explanation. Go build something with it.

Follow me for more real DevOps guides: Abdo Boshy

If this saved you time, a clap means more than you think.