Immutable Homelab: Production-Grade Patterns on a Single Proxmox Node

Chapter 6: GitOps at Scale (ArgoCD and the Art of Not Repeating Yourself)

Catching up? Read Chapter 5: Bootstrapping Kubernetes and the GitOps Handoff.

If you have ever managed a Kubernetes cluster, you know the exact moment the honeymoon phase ends. It is usually around application number twenty.

At first, deploying workloads is incredibly satisfying. But soon, your repository becomes an unmaintainable graveyard of duplicated YAML. You find yourself copy-pasting the same 50 lines of Service, Deployment, and IngressRoute manifests for every new tool you run. When you decide to shift your storage class, rotate your internal domain, or tweak an ingress annotation, you are suddenly grepping through dozens of directories, manually editing files, and praying you don’t introduce a typo.

This is the copy-paste anti-pattern, and it is how clusters rot.

In this chapter, we enter the Application Layer. We will explore the architectural patterns that allow my apps repository to manage over 60 workloads automatically. The goal here is strict: apply DRY (Don't Repeat Yourself) software engineering principles to Kubernetes configurations. We will build a system where adding a new service is a simple two-file operation, and updating global cluster policies takes exactly one change in one file.

(You can view the full GitOps codebase powering this in the Starktastic Homelab GitHub organization.)

Pattern 1: Automatic Service Discovery (The Matrix Generator)

In a naive GitOps setup, you maintain an “App-of-Apps” master manifest. Every time you add a service, you manually register it by writing a new ArgoCD Application manifest. This is manual labor disguised as automation.

To bypass this, my bootstrap pipeline relies on a highly advanced ArgoCD ApplicationSet driven by a Matrix Generator. It continuously scans my infrastructure and services directories, looking for any folder containing an app.yaml file.

But it doesn’t just discover them — it orchestrates them via a strict RollingSync strategy. If we apply all 60 apps simultaneously, the cluster sync devolves into a thundering herd — CRDs race their consumers, controllers haven't installed before their Custom Resources apply, and the API server gets hammered. Instead, the generator parses the deployPhase key from the app.yaml to ensure dependencies load in the correct order (CRDs → Foundation → Controllers → Services).

# bootstrap/appsets/cluster-apps.yaml (Snippet)
apiVersion: argoproj.io/v1alpha1
kind: ApplicationSet
metadata:
  name: cluster-apps
  namespace: argocd
spec:
  generators:
    - matrix:
        generators:
          - list:
              elements:
                - category: infrastructure
                  pathPattern: "infrastructure/**/app.yaml"
                  deployPhaseDefault: controllers
                - category: services
                  pathPattern: "services/**/app.yaml"
                  deployPhaseDefault: services
          - git:
              repoURL: [https://github.com/Starktastic-Homelab/apps.git](https://github.com/Starktastic-Homelab/apps.git)
              revision: HEAD
              files:
                - path: "{{ .pathPattern }}"
  strategy:
    type: RollingSync
    rollingSync:
      steps:
        - matchExpressions:
            - key: deploy-phase
              operator: In
              values: [crds]
        - matchExpressions:
            - key: deploy-phase
              operator: In
              values: [foundation]
        # ... controllers and services follow

If I want to deploy a new service, I don’t touch ArgoCD. I simply create a new folder under services/, drop an app.yaml inside it, and the Matrix Generator dynamically provisions the application into the correct deployment phase.

Pattern 2: Multi-Source Rendering via Template Patches

Once an application is discovered, how does it compile its configurations without copy-pasting standard boilerplate? We solve this by implementing Hierarchical Value Cascading combined with ArgoCD’s Multi-Source Rendering.

To accomplish this seamlessly across dozens of different applications, my ApplicationSet uses a templatePatch to dynamically assemble multiple source layers into a single application. It pulls the upstream Helm chart, maps our private Git repo as the $values source, and conditionally mounts the custom ingress chart:

# bootstrap/appsets/cluster-apps.yaml (templatePatch Snippet)
spec:
  sources:
    # Source 1: The Upstream Helm Chart
    - repoURL: '{{ dig "chart" "repo" .defaultChartRepo . }}'
      chart: '{{ dig "chart" "name" .defaultChartName . }}'
      targetRevision: '{{ dig "chart" "version" .defaultChartVersion . }}'
      helm:
        valueFiles:
          - $values/templates/globals.yaml
          - $values/{{ .commonValues }}
          - $values/{{ $valuesPath }}/values.yaml

    # Source 2: Our Git Repository (mapped as $values)
    - repoURL: [https://github.com/Starktastic-Homelab/apps.git](https://github.com/Starktastic-Homelab/apps.git)
      targetRevision: HEAD
      ref: values

    # Source 3: The custom Ingress Abstraction Chart
    - repoURL: [https://github.com/Starktastic-Homelab/apps.git](https://github.com/Starktastic-Homelab/apps.git)
      targetRevision: HEAD
      path: templates/ingress-chart

The cascading value structure works like this:

• **globals.yaml (The Universal Truth):** Holds cluster-wide parameters such as storage classes and network domains.
• **common.yaml (The Architectural Baseline):** Enforces standard security contexts and default probes.
• **values.yaml (The Delta):** The only file the operator writes, defining what makes this specific workload unique.

If we migrate our primary storage server, we modify a single string in globals.yaml. ArgoCD dynamically injects that file into the render pipeline for all 60 applications, executing a coordinated rolling update across the entire cluster.

Pattern 3: Base and Variant Inheritance (baseApp)

A common operational headache occurs when you need to deploy multiple instances of the same application.

Consider running Seerr, the request-management UI my household uses to browse and request shows. The base instance serves the family in English. For a Russian-speaking household member, I run a separate seerr-ru variant with its own UI language preference and its own request history. Normally this requires duplicating the entire configuration directory. In my architecture, we use Base/Variant Inheritance.

The variant points to the base via baseApp, inheriting the entire chart configuration, and only overrides what genuinely differs — in this case, just the ingress hostname so traffic routes to the right backend:

# services/media/seerr-ru/app.yaml
name: seerr-ru
namespace: media
deployPhase: services
baseApp: services/media/seerr

ingress:
  enabled: true
  host: "request-ru"

The entire configuration for the localized variant is less than 15 lines of YAML.

Seerr’s variant only needed a different hostname. But what if the variant needs to change the underlying Helm chart values too? Take qbittorrent-ru — a second torrent client running on the same node as the base. It needs its own bittorrent listen port (to avoid clashing with the base) and its own MetalLB LoadBalancer IP. Setting valuesOverride: true tells ArgoCD to layer the variant’s own values.yaml on top of the base’s, so I only have to declare the deltas:

# services/media/qbittorrent-ru/app.yaml
name: qbittorrent-ru
namespace: media
deployPhase: services
baseApp: services/media/qbittorrent
valuesOverride: true # also layer this variant’s own values.yaml

ingress:
  enabled: true
  host: qbittorrent-ru
  domainType: "internal"
  port: 8080
  serviceName: qbittorrent-ru-main
  auth: true
  rateLimit: true

# services/media/qbittorrent-ru/values.yaml — only the delta from the base
service:
 bittorrent:
   annotations:
     metallb.universe.tf/loadBalancerIPs: "{{ .Values.global.network.loadBalancers.qbittorrentRu }}"
   ports:
     bittorrent-tcp: { port: 56882 }
     bittorrent-udp: { port: 56882 }

Same inheritance pattern, one extra flag, and the variant gets its own network identity while still riding on the base’s chart, secrets, and persistence layout.

Pattern 4: Declarative Ingress & The 80/20 Rule

Writing raw Traefik IngressRoute manifests for every single application is highly prone to copy-paste errors. You have to manually map TLS secrets, wire up authentication middlewares, and attach security plugins.

To simplify this, I built a Custom Ingress Chart Abstraction (templates/ingress-chart). Instead of writing raw networking YAML, my application's local app.yaml defines its ingress requirements declaratively:

# services/operations/homepage-admin/app.yaml
ingress:
  enabled: true
  host: "admin"
  domainType: "public"
  port: 3000
  auth: true
  authAdmin: true
  rateLimit: true

When ArgoCD renders the application, the templatePatch automatically injects my custom ingress chart. The chart evaluates those flags and generates the complex Traefik networking on the fly. Based on a simple authAdmin: true flag, the system automatically routes traffic through my strict Authentik administrator pipeline. If it's exposed externally, CrowdSec is automatically attached.

(A deliberate trade-off: To keep our middlewares DRY, Traefik must be configured with allowCrossNamespace: true. While this reduces strict namespace isolation, it is a calculated choice that heavily streamlines operations in a homelab).

The 80/20 Escape Hatch: This ingress chart intentionally has limitations. It handles a single host and a single port. It perfectly covers 80% of my cluster’s workloads.

But what about the 20% that require complex path-based routing? Rather than overcomplicating the ingress template, I treat it as a deliberate trade-off. For complex workloads, I simply set manifests: true in the app.yaml. ArgoCD natively overlays any bespoke YAML files I place inside a local manifests/ directory for that specific app. This gives me the speed of an abstraction layer with the infinite flexibility of raw Kubernetes YAML.

Pattern 5: The Offline Sealed Secrets Loop

Managing sensitive data is the ultimate GitOps challenge. One must not store plaintext secrets in Git, especially not in a public repo.

We solve this using Bitnami’s sealed-secrets. But we go a step further by integrating it directly into our bootstrap loop, ensuring that secrets can be encrypted offline before the cluster even exists.

Because Ansible pre-seeds the cluster with a vault-backed, deterministic TLS keypair during the Day-0 bootstrap phase (chapter 5), my local seal.sh utility can encrypt secrets on my workstation using the public certificate of that future cluster.

Because this file (sealed-secrets-cert.pem) is strictly a public RSA certificate, it is completely safe to commit directly to my public GitHub repository. When the new cluster is built and the sealed-secrets controller initializes, it adopts the pre-seeded private key and immediately materializes each committed SealedSecret into its corresponding plaintext Secret.

Pattern 6: Defensive Engineering & Survivability

When you rely entirely on automation, you must engineer your workloads to survive unexpected failures. A truly “immutable” architecture anticipates chaos.

The Postgres Zombie Lock

When you run persistent state — like a relational database — on top of an NFS mount, you eventually hit reality. During a sudden hypervisor hard crash, my PostgreSQL database lost power and failed to execute its standard shutdown sequence. It left behind an orphaned postmaster.pid lock file on the TrueNAS NFS volume.

When the cluster recovered, Postgres saw the leftover lock file, assumed another instance was already running, and entered an infinite crash loop.

I fixed this by injecting a lightweight initContainer directly into the Postgres configuration. Before the database is ever allowed to start, a script checks the storage for zombie locks and clears them:

# infrastructure/controllers/databases/postgres/values.yaml
initContainers:
  - name: rm-postmaster-lock-file
    image: busybox:latest
    securityContext:
      runAsUser: 1001 # Matches fsGroup, avoiding root escalation
      runAsNonRoot: true
    command:
      - sh
      - -c
      - |
        if [ -f /bitnami/postgresql/data/postmaster.pid ]; then
          echo "🧟 Zombie Lock Found: Deleting postmaster.pid"
          rm -f /bitnami/postgresql/data/postmaster.pid
        fi

The CrowdSec Air-Gap & mTLS

Similarly, my external-facing ingress controller relies on a CrowdSec “bouncer” plugin to drop malicious traffic. Traditionally, Traefik downloads this plugin from GitHub at runtime. But if my node restarts and GitHub is experiencing an outage, Traefik fails to boot. To solve this, I pre-fetch the CrowdSec bouncer via an initContainer into an emptyDir on startup, surviving network outages.

Additionally, many operators use password-based auto-registration for their CrowdSec agents. When a pod restarts, the agent attempts to register again, fails with a “user already exists” error, and crashes. To fix this fragility, I integrated cert-manager to establish Mutual TLS (mTLS) between the agent and the central API. Cert-based auth means the agent identifies itself by its cert’s CN on every connect — there is no separate registration step to fail on restart.

Pattern 7: The Merge-Driven CI/CD Engine

Since our infrastructure is merge-driven, no engineer ever logs into the cluster to apply changes manually. Every alteration starts as a Pull Request. But reviewing a YAML PR is notoriously difficult. To make this safe, I built a scope-aware CI/CD engine in GitHub Actions that operates in two phases.

Phase 1: The Pull Request Gate (validate-and-diff.yml)

Before a PR can be merged, it must pass a strict 3-tier validation pipeline:

1. YAML Linting: Ensures syntax is flawless using yamllint.
2. Schema Validation: Uses kubeconform to validate against official Kubernetes schemas plus the community-maintained datreeio/CRDs-catalog for our custom resources.
3. ArgoCD Diff Previews: How does the CI know which applications to dry-run when I modify a shared file? The cluster-apps.yaml ApplicationSet dynamically injects a broad glob annotation into every application:

annotations:
  argocd-diff-preview/watch-pattern: '{{.path.path}}/.*, templates/.*'

When the runner executes, it spins up a throwaway, in-runner ArgoCD instance, renders both main and the PR branch through it, and diffs the resulting manifests. Because of the watch-pattern, if I update services/media/qbittorrent/values.yaml, the pipeline generates a diff exclusively for qBittorrent. But if I update templates/globals.yaml, the workflow automatically recalculates the structural diff across all 60 affected applications and posts it as an inline PR comment.

Phase 2: The Post-Merge Sync (refresh.yaml)

ArgoCD’s controller polls Git every three minutes by default, so a merged PR would eventually converge on its own. But three minutes is an eternity when you’re iterating on a config change and want to see the result before your coffee cools. This post-merge workflow closes that loop in real time: the moment a PR lands on main, it fires a targeted refresh so the affected applications reconcile within seconds instead of minutes.

To avoid hammering the API server with a blanket “refresh everything” on every commit, a bash script intelligently determines the scope. It walks up the directory tree to identify exactly which applications were modified.

If it detects changes to specific apps, it passes those names to the ArgoCD CLI and executes a lightning-fast, 10-way parallel refresh across the cluster:

refresh_apps() {
  echo "$@" | tr ' ' '\n' | xargs -P 10 -I {} sh -c '
    argocd app get "$1" --core --refresh > /dev/null && echo "✓ $1" || echo "✗ $1"
  ' _ {}
}

A Note on Homelab vs Production

While these patterns are production-grade, it is important to acknowledge that this is still a homelab. In a true enterprise environment, this repository would feature strict NetworkPolicies to isolate namespaces and PodDisruptionBudgets (PDBs) to ensure quorum during node drains.

The Moment of Truth

At this point, our entire infrastructure is defined. We have built the hypervisor network, baked the golden images, provisioned the virtual hardware, bootstrapped the cluster, and scaled our application deployment to a highly robust, DRY-compliant GitOps pipeline.

But there is one final, critical question. We claim our infrastructure is entirely declarative and disposable. We claim that we don’t treat our servers like pets.

Next week, we put that claim to the ultimate test. We will execute the Meteor Test: we will delete the entire cluster, and watch it rebuild itself from absolute zero to a fully recovered, operational state in less than fifteen minutes.