I Built a Production CI/CD Pipeline with Harness + GitHub Actions + ArgoCD. Here’s What I Learned

There is a well-worn trap in DevOps: building a CI/CD pipeline that works, then stopping there. The pipeline runs. The app deploys. Job done.

But a pipeline that only deploys is missing most of what matters in production — security scanning before images reach the cluster, policy enforcement before manifests are merged, approval gates before production, automated rollback when things go wrong, and an audit trail for every change.

In this post, I will walk through a reference implementation I built for a seven-service Node.js microservices application on Kubernetes. The goal was not just a working pipeline, but a production-representative one: something that demonstrates the full delivery lifecycle including PR gates, image scanning, SBOMs, OPA policy enforcement, approval governance, rolling and canary deployments, GitOps, and secrets management.

All the code is open source: github.com/gitafolabi/harness-cicd

The Application

The app is a boutique e-commerce platform with seven services:

-Auth — JWT-based authentication (Node.js, port 3002)

• Gateway — API gateway and routing (Node.js, port 3001)
• Orders — order management (Node.js, port 3005)
• Product Service — product catalogue (Node.js, port 3003)
• User Service — user management (Node.js, port 3006)
• Notification Service — RabbitMQ-backed email/event notifications
• Frontend — React frontend

Seven services mean seven Dockerfiles, seven Kubernetes manifests, seven CI build jobs, and seven CD stages. It is large enough to demonstrate real patterns without becoming noise.

Why Three Tools

The most common question I get about this setup is: “Why use GitHub Actions and Harness and ArgoCD? Isn’t that overkill?”

It is the opposite, as each tool does something the others don’t.

GitHub Actions excels at CI because it runs close to the developer. Every pull request triggers linting, policy checks, and Dockerfile validation in seconds. When code merges, GHA builds the images in parallel with layer caching (type=gha) and pushes to Docker Hub.

Harness takes over from there. It adds what GHA lacks in the CD space: a proper approval gate with RBAC (the engineer who triggered the pipeline cannot be the only approver in a real SOC 2 shop), automatic rollback if the deploy fails a readiness probe, and a full audit trail of who deployed what SHA to which environment at what time.

ArgoCD handles everything that should always be reconciled, like ingress controllers, ESO, and monitoring agents. It never sleeps. If someone manually patches the cluster, ArgoCD corrects it on the next sync cycle.

The Full Flow

Developer push → GitHub PR ▼ PR Gate (GHA) • OPA/Conftest • yamllint • hadolint │ merge to main ▼ CI (GHA) • Docker build • Trivy image scan • SBOM (Syft) • Push SHA + latest │ Harness REST API ▼ CD (Harness) • Approval gate • Rolling deploy • Auto-rollback ▼ Kubernetes — boutique ns

Security Gates: Defence in Depth

Security is applied at three layers, each catching a different class of problem.

Layer 1 — PR Gate (before merge)

Three jobs run in parallel on every pull request:

OPA/Conftest — policy-as-code for Kubernetes manifests. I wrote Rego policies that enforce:

• Every container must define resources.limits
• No container may run as root (runAsUser: 0)
• “allowPrivilegeEscalation” must not be true
• “readOnlyRootFilesystem” must not be explicitly ‘false.’
• Services must not use NodePort

deny[msg] {
  input.kind == "Deployment"
  container := input.spec.template.spec.containers[_]
  not container.resources.limits
  msg := sprintf(
    "Deployment '%s': container '%s' is missing resource limits.",
    [input.metadata.name, container.name],
  )
}

This is enforcement, not advisory. A missing ‘resources.limits’ blocks the merge.

Layer 2 — CI Image Scan (after build, before push)

After each image is built, Trivy scans it against the CVE database before it’s ever pushed to Docker Hub:

- name: Trivy image scan (block on CRITICAL/HIGH)
        uses: aquasecurity/trivy-action@master
        with:
          image-ref: ${{ matrix.image }}:${{ github.sha }}
          format: table
          exit-code: "1"
          ignore-unfixed: true
          vuln-type: os,library
          severity: CRITICAL

‘ignore-unfixed: true’ is important as it prevents blocking on CVEs that have no fix yet. There’s no point failing the pipeline on a vulnerability you can’t act on.

Layer 3 — SBOM Generation

After the image scan, Syft generates a Software Bill of Materials in CycloneDX JSON format and uploads it as a GitHub Actions artifact

- name: Generate SBOM (Syft)
        uses: anchore/sbom-action@v0
        with:
          image: ${{ matrix.image }}:${{ github.sha }}
          format: cyclonedx-json
          output-file: sbom-${{ matrix.service }}-${{ github.sha }}.json
          upload-artifact: true
          artifact-name: sbom-${{ matrix.service }}-${{ github.sha }}

The value is not obvious until a CVE is disclosed. When Log4Shell dropped, organisations with SBOMs knew within hours which services were affected. Without them, they were scanning every image manually. The NTIA minimum elements and the EU Cyber Resilience Act are making SBOMs mandatory for software sold to government, thereby building the habit now is the right move.

Kubernetes Hardening

Every application manifest follows the same security context pattern:

spec:
  securityContext:
    runAsNonRoot: true
    runAsUser: 1001
  containers:
    - name: auth
      securityContext:
        readOnlyRootFilesystem: true
        allowPrivilegeEscalation: false
        capabilities:
          drop: [ALL]
      volumeMounts:
        - name: tmp
          mountPath: /tmp
  volumes:
    - name: tmp
      emptyDir: {}

‘readOnlyRootFilesystem: true’ prevents an attacker who gains code execution from writing malware to disk. Node.js apps need ‘/tmp’ writable, so an ‘emptyDir’ volume is mounted there — a per-pod ephemeral scratch space that does not open the whole filesystem.

‘capabilities: drop: [ALL]’ strips all Linux capabilities from the container process. Even if the process runs as non-root, Linux capabilities (‘NET_ADMIN’, ‘SYS_PTRACE’, etc.) can still grant elevated privileges. Dropping all of them removes that attack surface.

‘runAsUser: 1001’ ensures the process runs as a named non-root user created in the Dockerfile:

RUN addgroup -g 1001 -S nodejs && adduser -S nodejs -u 1001
USER nodejs

The Harness Concepts

The Delegate

The Harness Delegate is the most important concept to understand. It is a worker agent you install inside your cluster. Harness SaaS communicates with it over an outbound HTTPS connection, as your cluster never receives inbound connections from Harness.

Every connector (GitHub, Docker Hub, Kubernetes) routes through the Delegate. This means cluster credentials, registry tokens, and Git PATs never leave your infrastructure. When the Delegate runs a ‘kubectl apply’, it uses its own in-cluster ServiceAccount, and no kubeconfig is serialised anywhere.

Services, Environments, and Infrastructure Definitions

These three concepts form Harness’s deployment model:

• Service = what you deploy. It bundles the artifact source (Docker Hub image) and manifest source (Git path).
• Environment = where (production, staging, dev).
• Infrastructure Definition = the specific target within an environment: which cluster connector, which namespace, which Helm release name.

In this project, all seven boutique services share one Environment and one Infrastructure Definition. They differ only in their service, which is different image paths and different manifest paths.

The Approval Gate

- step:
    type: HarnessApproval
    timeout: 1d
    spec:
      approvers:
        userGroups:
          - Boutique_Approvers
        minimumCount: 1
        disallowPipelineExecutor: false

A few things that tripped me up

The user group must be project-scoped, not account-scoped. Account-level groups (‘account.All Account Users’) are rejected by project-level pipeline approval steps. I created ‘Boutique_Approvers’ as a project group.

disallowPipelineExecutor: false’ allows the pipeline trigger to also be the approver, which is appropriate for a demo environment. In production, set this to ‘true’ to enforce separation of duties (a SOC 2 requirement).

Why ‘tag: latest’ in the CD stages?

I built two pipeline scenarios. In Scenario A (Harness CI+CD), the image tags use ‘<+pipeline.sequenceId>’, which is a unique integer per execution. In Scenario B (GHA CI + Harness CD), the CD pipeline receives the git SHA as a variable: ‘<+pipeline.variables.imageTag>’.

When retrying a pipeline from the approval stage (re-deploying the same build without rebuilding), the sequenceId approach fails as the retry gets a new sequenceId, but no image was pushed with that tag. Pushing both SHA and ‘latest’ during CI solves this. The ‘latest’ tag always points to the most recent successful build.

Two Pipeline Scenarios and Why Scenario B is the Active One

The repo includes both approaches so readers can choose what fits their team. But there is a clear winner for most organisations, and it is worth explaining why.

Scenario A — Full Harness CI+CD (link):

Harness owns everything. CI uses Kaniko on KubernetesDirect as it builds container images inside your cluster without needing Docker socket access. This is a single-platform story: one tool, one audit trail, one place to look.

The tradeoffs become apparent quickly. On a small cluster (two nodes), seven parallel Kaniko builds exhaust CPU, and pods sit ‘Pending’. The builds must run sequentially as seven services × ~2 minutes each = 14+ minutes before a single image reaches the registry. Harness Cloud (hosted runners) would avoid this, but it requires a credit card even on the free tier.

More fundamentally, Harness CI is powerful, but it is not where most developer workflows live. GitHub already has the PR, the review, the code context, and the status checks. Building CI outside GitHub means developers check two places for build feedback instead of one.

Scenario B — GitHub Actions CI + Harness CD (link):

This is the active pipeline. GHA runs CI, and Harness runs CD. The split is deliberate.

GHA builds all seven images in parallel with Docker layer caching (‘type=gha’) seven images in approximately four minutes. It scans each image with Trivy, generates an SBOM with Syft, and pushes two tags to Docker Hub: the git SHA for traceability and ‘latest’ for retry resilience. All of this runs natively inside GitHub, where the developer already is.

When all builds pass, GHA calls the Harness REST API to hand off to CD:

curl -X POST \
  "https://app.harness.io/gateway/pipeline/api/pipeline/execute/boutique_full_cicd?accountIdentifier=...&orgIdentifier=default&projectIdentifier=BoutiqueApp" \
  -H "x-api-key: $HARNESS_API_KEY" \
  -H "Content-Type: application/yaml" \
  --data "pipeline:
  identifier: boutique_full_cicd
  variables:
    - name: imageTag
      type: String
      value: $GITHUB_SHA"

The git SHA flows through every CD stage as ‘<+pipeline.variables.imageTag>’. Harness deploys exactly what GHA built and there is no ambiguity about which image version is in production.

Why is the CD side Harness and not just more GHA?

GitHub Actions can deploy to Kubernetes. So why bring in Harness at all?

Because deployment is not the hard part. Governance is the hard part.

GHA has no native concept of a multi-step approval gate backed by RBAC. It has no automated rollback that triggers on a failed readiness probe. It has no deployment verification step that queries Prometheus and rolls back if the error rate spikes. It has no per-environment audit trail showing who approved SHA ‘f9483b9’ into production at 14:32 on Tuesday.

These are not nice-to-haves in a regulated or high-stakes environment. They are table stakes. Harness was built specifically for this layer. GitHub Actions was built for automation. Using each for what it was designed for is not over-engineering, it is the right tool selection.

The practical result: developers get fast feedback in GitHub (PR gate in ~30 seconds, CI build in ~4 minutes), and the platform team gets the governance controls they need in Harness (approval chains, rollback, DORA metrics, audit log) without compromising either.

Canary Deployments

A canary pipeline is included for the API Gateway (link). It demonstrates the two-phase pattern: Phase 1 — Canary (20%)

K8sCanaryDeploy → 20% of pods get the new version

HarnessApproval → reviewer validates error rate and p99 latency

[production: replace with Harness SRM Continuous Verification]

Phase 2 — Primary (100%)

K8sRollingDeploy → promotes new version to all pods

The approval step can be replaced with a Harness SRM ‘ContinuousVerification’ step, which queries Prometheus or Datadog and compares canary metrics against the primary baseline automatically. If error rate increases by more than your threshold, Harness rolls back without human intervention.

When to use each strategy:

Secrets Management

The secret architecture has three layers: Cloud Secret Store (Azure Key Vault / AWS SM / GCP SM / Vault) │ External Secrets Operator (sync every 1h) ▼ Kubernetes Secret: boutique-secrets │ secretKeyRef in pod spec ▼ Application pods

The critical lesson: never ‘kubectl patch’ a secret that ESO manages. ESO overwrites any manual change on its next sync cycle. Always add the secret to the cloud vault first, add an entry to the ‘ExternalSecret’ resource, then force a sync:

kubectl annotate externalsecret boutique-secrets \
  force-sync=$(date +%s) --overwrite -n boutique

GitOps with ArgoCD and Harness

ArgoCD manages the platform layer, like the ingress controller, External Secrets Operator, and monitoring. Harness manages the application layer, which is the boutique services with an explicit pipeline and approval gate.

Connecting ArgoCD to Harness GitOps gives unified visibility: both the ArgoCD app sync status and the Harness CD pipeline executions appear in one dashboard. Harness RBAC controls who can trigger ArgoCD syncs, rather than relying on ArgoCD’s own RBAC configuration.

For teams wanting fully GitOps-driven progressive delivery, Argo Rollouts is the natural next step: a ‘Rollout’ CRD replaces the standard ‘Deployment’ and manages canary traffic splitting natively, driven entirely by Git commits via ArgoCD.

Kubernetes Providers

Everything in this repo, except the connector file, is provider-agnostic. The Harness Delegate installs via Helm on any Kubernetes cluster. The auth mechanism is the only thing that varies:

OpenShift’s Security Context Constraints are stricter than standard Kubernetes PodSecurityAdmission. The security contexts in this repo (‘readOnlyRootFilesystem: true’, ‘capabilities: drop: [ALL]’) are compatible with the OpenShift ‘restricted-v2’ SC, so no manifest changes needed.

Lessons Learned

1. The Delegate is the foundation; get it running first. Every pipeline failure I had in the first hour was either a missing Delegate or a misconfigured connector. Verify the Delegate shows “Connected” before writing a single pipeline stage.

2. Scope matters in Harness RBAC. Account-level entities (user groups, connectors) cannot be referenced by project-level resources without explicit sharing. I spent an hour debugging an approval gate failure that turned out to be an account-level user group being used in a project-level pipeline.

3. ‘repoName’ is required when the connector uses an Account URL. The GitHub connector can be configured with an Account URL ([https://github.com/myorg](https://github.com/myorg`)) or a Repository URL. If you use Account URL, every manifest store spec in every Service YAML must include ‘repoName’. Missing it gives ‘repo name cannot be empty’ error mid-pipeline.

4. ‘ignore-unfixed: true’ in Trivy is not optional. Without it, every pipeline fails on CVEs in transitive dependencies that have no available patch. The pipeline blocks on noise instead of actionable findings.

5. OPA policies are for your rules, not industry benchmarks. Trivy checks against CIS and NSA hardening benchmarks. OPA/Conftest enforces your team’s specific standards. Both are valuable, but they serve different purposes.

If you are learning Harness, building a platform engineering function, or preparing for a DevOps role that covers this stack. I hope it is useful.