Goodbye ClickOps: Building GCP Load Balancers the Kube-Native Way

Author: Shivam Narula

At Groww, our infrastructure runs on Google Cloud Platform (GCP) and is managed declaratively using Crossplane. As our platform grew, our engineering teams needed to provision load balancers across a wide range of configurations: external and internal, global and regional, application and network.

There is no single “LoadBalancer” resource in either GCP or Crossplane. A GCP load balancer is a logical concept composed of several discrete resources like forwarding rules, target proxies, URL maps, backend services, and health checks. Crossplane exposes these as individual managed resources with no higher-level abstraction out of the box.

We solved this gap by building a single Crossplane Composition called xIdpLoadBalancer. This exposes a clean API and generates the correct GCP resources for any topology. This post walks through how it works, the design decisions behind it, and the patterns that made dynamic resource generation possible.

Crossplane in 60 Seconds

If you have not worked with Crossplane before, here are the core concepts:

| Concept                           | Kind                        | What it is                                                             | Analogy                                                         |
|-----------------------------------|-----------------------------|------------------------------------------------------------------------|-----------------------------------------------------------------|
| CompositeResourceDefinition (XRD) | CompositeResourceDefinition | Defines the schema for a custom resource.                              | Like a CRD, but for platform APIs.                              |
| Composition                       | Composition                 | A template that maps a composite resource's spec to managed resources. | A CloudFormation template, but in Kubernetes.                   |
| Composite Resource (XR)           | e.g. xIdpLoadBalancer       | An instance of a type defined by an XRD. Holds the full desired state. | The internal work order.                                        |
| Claim                             | e.g. IdpLoadBalancer        | A namespace-scoped handle to an XR. This is what teams apply.          | A customer-facing ticket that maps to an internal work order.   |
| Managed Resource (MR)             | e.g. BackendService         | A Crossplane resource representing a single external API object.       | A Terraform resource.                                           |
| go-templating function            | Composition pipeline step   | Renders managed resources from Go templates using the XR's spec.       | A Helm chart engine running inside the Crossplane control loop. |

In our system, teams interact only with the IdpLoadBalancer Claim via a Helm chart. Everything below that line is owned and maintained by the platform team.

The Problem with Raw Managed Resources

A GCP load balancer is a graph of resources. The exact resources and their kinds differ depending on the configuration:

| Dimension    | Options               | Effect                                          |
|--------------|-----------------------|-------------------------------------------------|
| type         | application / network | HTTP(S) vs TCP proxies, URL map vs not.         |
| access       | external / internal   | Load balancing scheme, IP type.                 |
| distribution | multi / single        | Global vs Regional resource kinds.              |
| classic      | true / false          | EXTERNAL vs EXTERNAL_MANAGED scheme.            |

Without a Composition, each team had to know exactly which combination of API resources to stitch together. With one, they specify a few high-level fields, and the platform handles the rest.

The Composition: Dynamic Resource Generation

Our Composition uses Crossplane’s go-templating function. At the top of the file, we define boolean variables from the composite’s spec to keep the logic readable and avoid deep nesting.

{{ $isGlobal := eq $ocr.spec.distribution "multi" }}
{{ $isRegional := eq $ocr.spec.distribution "single" }}
{{ $isExternal := eq $ocr.spec.access "external" }}
{{ $isInternal := eq $ocr.spec.access "internal" }}

These variables thread through every resource block, cleanly selecting the correct GCP resource kind (like choosing between a GlobalAddress and a regional Address).

The Array Index Trap: A Hard Lesson Learned

Dynamic generation requires looping over arrays of user-defined backends and health checks to create resources. This introduced a massive engineering trap.

Every managed resource emitted by a Crossplane template must carry a unique, stable tracking annotation called gotemplating.fn.crossplane.io/composition-resource-name. This is how Crossplane tracks identity across continuous reconciliation loops.

Our initial instinct was to use the array index ($index) to generate this name. This is a catastrophic mistake.

Consider a load balancer with 30 health checks. If a user removes the entry at index 10, every health check from index 11 onwards shifts left by one position in the array. Crossplane sees a different name at each position and attempts to reconcile each shifted resource toward its new configuration.

In practice, health check 11’s configuration gets written over health check 10’s live GCP object. The updates are completely silent, propagating wrong configurations to live backend services.

The Fix: Stable Identifiers

Composition resource names must be derived from a stable, user-provided identifier rather than an array position. We introduced a mandatory nameOverride field.

Here is the wrong way:

metadata:
  annotations:
    crossplane.io/external-name: {{ $healthCheck.nameOverride }}
    gotemplating.fn.crossplane.io/composition-resource-name: {{ $name }}-hc-{{ $index }}  # Unstable
  name: {{ $name }}-hc-{{ $index }}

Here is the right way, using a fail guard:

{{- if empty $healthCheck.nameOverride }}
{{- fail "nameOverride is required for each healthCheck" }}
{{- end }}
metadata:
  annotations:
    crossplane.io/external-name: {{ $healthCheck.nameOverride }}
    gotemplating.fn.crossplane.io/composition-resource-name: {{ $healthCheck.nameOverride }}  # Stable
  name: {{ $healthCheck.nameOverride }}

If nameOverride is absent, the Composition fails at render time. An early error is infinitely better than silent configuration drift.

Dynamic Backend Service Generation

Backends follow the same pattern, dynamically adapting to the user’s requested scope:

{{- range $index, $backend := $ocr.spec.backendConfiguration.backends }}
---
{{- if eq $backend.type "bucket" }}
kind: BackendBucket
{{- else if $isGlobal }}
kind: BackendService
{{- else if $isRegional }}
kind: RegionBackendService
{{- end }}

Advanced Features Built for Scale

Importing Existing Load Balancers (Observe Mode)

Migrating legacy infrastructure is risky. By toggling a simple lb.existing: true flag in our Helm chart, the Composition switches into an observe-only mode. It generates stub managed resources that reference existing GCP names via crossplane.io/external-name. Crossplane monitors their state without ever attempting to create, update, or delete them, allowing for zero-downtime migrations.

Independent Backend Management Policies

Often, backend services are owned by entirely different teams than the load balancer itself. We added backendManagementPolicies to provide an override. The load balancer frontend can be fully managed (Create, Update, Delete), while the backends are set to Observe only. This creates safe, explicit ownership boundaries.

Shifting Validation Left

Crossplane validates inputs at reconciliation time, meaning errors only surface after the resource is applied to the cluster. We pushed validation earlier by utilizing Helm’s fail function in the composite template. If a developer tries to mix incompatible settings, the deployment fails instantly during the local helm template dry run.

Conclusion

Abstracting infrastructure is not just about writing less code. It is about removing cognitive load and preventing systemic errors. By defining a single smart Crossplane Composition, we empowered our product teams to spin up highly complex, production-ready GCP Load Balancers securely, eliminating the risks of manual ClickOps and creating a true Kubernetes-native infrastructure platform.