Now the Infrastructure Is Boring Too (And That’s Still a Compliment)

This is a follow-up to How We Made Deploying a New Service Boring — and That’s a Compliment. If you haven’t read it yet, the short version: we replaced dozens of per-service Helm charts with one shared chart (zc-monochart) and made deployment of a new service a non-event. This post is about what we did next.

The problem we didn’t solve

After monochart landed, deploying a new service became genuinely boring. A developer creates .infra/values.yaml, pushes a branch, and the CI pipeline handles the rest. Done.

But deploying a service is only half the story.

Almost every service we run needs something from AWS. An SQS queue to process background jobs. A PostgreSQL database. A Secrets Manager secret. An S3 bucket. Sometimes several of these at once. And for all of that, the workflow looked like this:

1. Developer adds a Terraform module block to the infrastructure repo and opens a PR
2. DevOps reviews the Terraform plan and sometimes helps with edge cases. The shared modules handled the common patterns, but corner cases still needed a second pair of eyes
3. CI runs, plan reviewed, PR merged
4. Terraform applies
5. Developer gets the queue URL back via Slack, hard-codes it into their config

For example, adding an SQS queue meant writing something like this, in a separate repo owned by a different team:

# tf-survey/us-east-1/13-sqs.tf  ← separate repo, separate team, separate context
module "sqs_my_new_queue" {
  source                     = "terraform-aws-modules/sqs/aws"
  version                    = "4.3.1"
  name                       = "MyNewQueue"
  visibility_timeout_seconds = "5400"
  create_dlq                 = true
  tags                       = merge(local.global_tags, { "Name" = "MyNewQueue-sqs" })
}

This worked. But it had two problems. First, it pulled DevOps into routine work that a developer could reason about themselves. Second, and more fundamentally, a service’s truth was now split across two repos. Deployment config in the service repo. Infrastructure config in the Terraform repo. Understanding a service fully meant context-switching between two codebases.

The irony wasn’t lost on us. We had made deployments boring. But the infrastructure side of the same service lived somewhere else entirely.

The insight

There’s a question that kept nagging at us: where does a service live?

The deployment side had a clean answer. After monochart, everything a service needs to run lives in .infra/values.yaml: replicas, resources, environment variables, ingress rules, autoscaling. One place: an .infra/ folder with a shared base config and small per-environment overrides, owned by the team building the service.

But the infrastructure side had a different answer. The SQS queue a service depends on? Terraform, in a central infrastructure repo. The database? Same. The Secrets Manager secret? Same. And so on for every AWS resource the service touches. A developer who wanted a full picture of their service had to look in at least two completely separate places (the service repo and the infra repo) and mentally stitch them together.

So the goal became clear: all the configuration that a service needs should live in one place, together. The deployment and the infrastructure. One place, one repo, one source of truth: a shared base config with small per-environment overrides.

This pointed us toward a clean model: monochart becomes the developer-facing API for everything a service needs, both to run and to be provisioned.

Enter Crossplane

Crossplane is a Kubernetes-native framework for provisioning and managing cloud infrastructure and APIs. The core idea: you define custom Kubernetes resources (called Composite Resources, or XRs) that represent infrastructure, and Crossplane reconciles them into real AWS (or any cloud) resources.

What makes this powerful is the abstraction layer. You write CompositeResourceDefinitions(XRDs) that define the API: what parameters a team can specify. You write Compositions that define the implementation: what AWS resources actually get created. Consumers only see the API; the implementation is hidden.

We started with four XRDs, covering the most common needs:

• AppWorkloadSqs -provisions an SQS queue, optionally with a DLQ
• AppWorkloadDatabase - creates a PostgreSQL database on the shared RDS instance
• AppWorkloadSecret -creates an AWS Secrets Manager secret
• AppWorkloadRole - creates an IAM role with EKS Pod Identity association

The model is extensible by design. Adding a new resource type means writing a new XRD and Composition, with no changes required to any service. These are our contracts. A developer declares what they need; Crossplane handles how it gets created.

In practice, everything lives in our GitOps repo under a single crossplane/ folder. Each resource type gets its own folder: an XRD that defines the API, and a Composition that implements it:

crossplane/
├── compositions-appset.yaml
├── providers-appset.yaml
├── providerconfigs-appset.yaml
└── app-resources/
    └── compositions/
        └── templates/
            ├── sqs/
            │   ├── xrd-app-workload-sqs.yaml
            │   └── composition-app-workload-sqs.yaml
            ├── database/
            ├── secret/
            ├── bucket/
            └── iam/
                ├── xrd-app-workload-role.yaml
                ├── composition-app-workload-role.yaml
                ├── policy-sqs.yaml
                ├── policy-s3.yaml
                ├── policy-secrets.yaml
                ├── policy-dynamodb.yaml
                └── ...

The *-appset.yaml files are ArgoCD ApplicationSets — they tell ArgoCD to deploy the compositions, providers, and provider configs to every cluster automatically. If you're coming from Terraform, think of providerconfigs as the equivalent of the provider "aws" {} block (credentials, region), and providers as required_providers.

Inside compositions/templates/, each resource type gets its own folder with two files: the XRD and the Composition. The XRD is the contract — it defines what parameters a developer can pass (like variables.tf in a Terraform module). The Composition is the implementation — it describes what AWS resources actually get created (like the module's main.tf). The iam/ folder also holds one policy file per resource type, which is the actual IAM JSON that gets attached to the role (the equivalent of aws_iam_policy_document in Terraform).

ArgoCD reconciles this folder continuously and deploys everything to every cluster. Adding a new resource type means adding a new folder here — no changes needed anywhere else.

Connecting the two halves

This is where monochart comes back in. We added templates to the chart that do the bridging work. When Helm deploys a service, it now also generates the Crossplane resources that service declared it needs:

# Developer writes this in .infra/values.yaml
sqs:
  enabled: true
  items:
    - name: tasks
      dlq: true
      maxReceiveCount: 5
    - name: notifications
      fifo: true
      visibilityTimeoutSeconds: 300

When Helm deploys this service, it renders standard Kubernetes manifests, and some of those manifests are now Crossplane custom resources:

apiVersion: platform.zencity.io/v1alpha1
kind: AppWorkloadSqs
metadata:
  name: myservice-tasks
spec:
  parameters:
    queueName: zc-staging-myservice-tasks
    region: us-east-1
    env: staging
    project: organic
    fifo: false
    dlq:
      enabled: true
      maxReceiveCount: 5

Crossplane picks that up and creates the actual SQS queue in AWS. No tickets. No Terraform PRs. No Slack messages with queue URLs.

The same pattern works for any AWS resource the service needs: databases, secrets, S3 buckets, and more.

From a developer’s perspective, this is just YAML they already understand. They’re not learning Terraform, they’re not understanding Crossplane internals. They’re just adding a few lines to the same file they already manage. And whenever we add support for a new resource type, it slots into the same pattern: same file, same workflow, no new concepts to learn.

The IAM piece: policies derived, not declared

One thing we were deliberate about: we didn’t want developers specifying IAM policy ARNs or trust relationships. That’s where things go wrong.

Instead, monochart auto-derives the permissions a service needs based on what it declares. If sqs.enabled: true, the service gets the SQS policy attached. The same logic applies to every resource type created by Crossplane.

A helper function walks through every declared resource and builds up the permission list automatically:

{{- if .Values.sqs.enabled }}{{ $policies = append $policies "sqs" }}{{- end }}
{{- ... and so on for every resource type }}

The AppWorkloadRole XR is then created automatically, attaching only the policies the service actually needs and wiring up EKS Pod Identity so the service's pods can assume the role without managing any credentials.

The permissions behind each option are written by us in the Crossplane Compositions. When a developer enables SQS access, they get exactly the policy we defined for it. They choose what they need and We decide what each choice actually grants in AWS.

ecause policies are defined centrally, there are no more over-permissioned services getting admin access to things they barely touch. When we update a policy, every service using that resource type gets the change immediately, in every environment, without anyone doing anything extra. And if a developer needs a permission that doesn’t exist yet, they come to us — we add it once, and it’s available to everyone.

The architecture

Developer edits .infra/values.yaml                    
     |                                        
     | Helm / monochart renders               
     |
     +---> Kubernetes objects   +----------------+
     |                          |  Deployment    |
     |                          |  HPA           |<--------------+
     |                          |  Ingress / ... |               |
     |                          +----------------+               |
     |                                                           |
     +---> AWS resources        +-------------------------+      |
                                |  AppWorkloadSqs         |      |
                                |  AppWorkloadDatabase    |<--+  |
                                |  AppWorkloadRole / ...  |   |  |
                                +-------------------------+   |  |
                                                              |  |
zc-gitops repo  ----> Crossplane XRDs + Compositions ---------+  |
                                                                 |
monochart repo  ----> new monochart package ---------------------+

The developer touches one place. monochart, managed by DevOps and released as a shared Helm package, controls what gets rendered and Crossplane also managed by DevOps and deployed by ArgoCD to every cluster and control what gets provisioned in AWS. Both are owned centrally, so every service benefits from every update automatically.

What this actually changed

Picture a developer joining a new team at Zencity. They pick up a service, open the .infra/ folder, and they can understand it completely: how it deploys, what it depends on, what permissions it holds. No Terraform repo to dig through, no tribal knowledge required. Everything is right there.

That’s what we were actually building: not a set of tools, but a platform. Monochart owns the deployment side. Crossplane owns the infrastructure side. ArgoCD keeps everything consistent across environments. Each part does its job quietly, and together they give developers full ownership of their service from one place, while we stay in control of the standards and policies that govern all of them.

There’s a phrase from our previous post that still applies here: good infrastructure is not meant to be impressive. It’s meant to be invisible. A developer shouldn’t need to think about queue naming, IAM trust relationships, or which Terraform module to use. They should think about their service.

Monochart made deployment boring. Crossplane, plugged into monochart, made infrastructure provisioning boring too.

That’s still the compliment we were going for.