How I Designed a Secure Two-Tier Architecture on GCP (and What Broke Along the Way)

A field note from building FreshCart’s backend infrastructure — a grocery delivery app that doesn’t exist yet, on real cloud architecture.

Intro

FreshCart is a hypothetical grocery delivery app, a two tier application built on a single VM to serve customers in the early stage of the product. In this article, I take on the role of FreshCart’s newly hired Cloud Engineer, here to help build the infra that helps them go from 100 concurrent users to 4000.

The problem with a single VM

The fastest way to get FreshCart’s backend online is a single VM, a public IP, and port 80 open. It would work. It would also mean that one machine is simultaneously your compute, your public attack surface, and your single point of failure — three jobs that have no business being done by the same box.

If that VM goes down, FreshCart goes down. If it gets compromised, whoever’s inside has a direct line to whatever else you bolt on later — a database, a cache, internal tooling — because nothing was ever designed to keep them apart. And there’s no way to grow past it without re-architecting under pressure, usually during the traffic spike that made the growth necessary in the first place.

So I built FreshCart’s backend as a two-tier architecture instead: a private compute layer that the internet never touches directly, fronted by a load balancer that’s the only thing actually exposed. Everything below is how that came together, entirely from the CLI, and the two decisions in the middle that turned out to be harder than they looked on paper.

The architecture

FreshCart’s infrastructure splits into two independent paths that both live inside one VPC, but expose themselves to the public internet in completely different ways.

The backend path starts with an external HTTP load balancer as the only public entry point. It health-checks a backend service, which — after a mid-build change I’ll get into — fans out to two instances of a placeholder nginx server (standing in for FreshCart’s actual app, since the point of this build is the network around the app, not the app itself). Both instances live in private subnets with no public IP assigned to either one. Getting out to the internet for OS updates and package installs happens through Cloud NAT, which is explicitly scoped to only the subnets that need it — nothing broader.

The static site path is a completely separate load balancer, this one terminating HTTPS with a Google-managed certificate, sitting in front of a Cloud Storage bucket that serves FreshCart’s “coming soon” landing page. It has its own public IP, its own certificate, and its own custom domain — deliberately decoupled from the backend path, since a compute-serving load balancer and a bucket-serving one prove genuinely different things.

A few specific choices are doing real work here, not just filling out a checklist:

--no-address

This one flag is the actual guarantee that the backend is unreachable except through the load balancer. Everything else — subnet placement, firewall rules — reinforces it, but this is the line that would break “private” if it were missing.

--source-ranges=130.211.0.0/22,35.191.0.0/16

The firewall rule that lets traffic reach the backend doesn’t open the door to the internet — it opens it specifically to Google’s documented load balancer and health-check ranges. Everything else stays denied by default, which is the actual point of putting a VM in a private subnet in the first place.

A lifecycle rule on the storage bucket moves objects to a cheaper storage tier after 30 days and deletes them after 90 — more on why that one required more thought than it should have, below.

[Insert your draw.io architecture diagram here — two lanes, one shared VPC, public/private boundary clearly marked]

The two hardest decisions

1. Adding real redundancy

I decided FreshCart’s backend shouldn’t depend on a single VM staying healthy. The fix seemed simple: a second private subnet, a second VM in a different zone, added as a second backend to the same load balancer.

Creating that second VM immediately failed with ZONE_RESOURCE_POOL_EXHAUSTED. That's not a misconfiguration — it's Google telling you, plainly, that the specific machine type you asked for doesn't have capacity in that specific zone right now. I tried other zones in the same region. Same error. I tried a smaller machine type. Same error, in every zone I tried.

The actual fix was switching the second VM to spot provisioning:

--provisioning-model=SPOT \
--instance-termination-action=STOP

Spot capacity draws from a separate pool than on-demand instances, which is exactly why it worked when nothing else did. The honest tradeoff: Google can reclaim that capacity with about 30 seconds’ notice, and --instance-termination-action=STOP means the VM stops rather than getting deleted outright — the disk and config survive, and it's one command to bring back. For a demo VM under active development, that's a complete non-issue. It would be a very different conversation for something serving real traffic.

What actually made this worth doing: adding the second VM only required touching few parts of the build stages. The load balancer’s health check, URL map, and forwarding rule never changed — they’re attached to the backend service, which now just fans out to two backends instead of one. If either zone has a problem, the health check stops routing to it and everything shifts to the survivor automatically.

2. The lifecycle rule that could delete my own live site

I had a simple requirement for the lifecycle rules: objects older than 30 days move to a cheaper storage tier (Nearline), and anything older than 90 days gets deleted. That was easy to implement as a JSON config:

{
  "rule": [
    {
      "action": {"type": "SetStorageClass", "storageClass": "NEARLINE"},
      "condition": {"age": 30}
    },
    {
      "action": {"type": "Delete"},
      "condition": {"age": 90}
    }
  ]
}

Problem is this rule applies to the entire bucket — and that same bucket is serving the live index.html for FreshCart's static site. Taken literally, day 90 doesn't just clean up old data. It deletes the file the site depends on to exist.

I had two real options: scope the rule to a subpath (something like an archive/ prefix) so it only ever touches content that's genuinely meant to age out, or apply it bucket-wide and accept that it's the easy way out while being wrong for a bucket that also serves live content.

I went with applying it bucket-wide, for a specific reason: trying to manage my GCP credits, the project will be get torn down within days, nowhere near the 30- or 90-day thresholds, so the conflict never actually matters here. But I don’t think that’s a reason to pretend the conflict doesn’t exist. If FreshCart’s static site were real and long-lived, I’d split this into two buckets — one serving live content with no lifecycle rule at all, and a separate one for anything that’s actually meant to age out.

What I’d add next

The obvious next step for a system like this — a second backend instance for redundancy — is already done. Asides that, here’s what a real, growing FreshCart would need that this capstone deliberately doesn’t have yet:

• Managed instance groups instead of unmanaged ones. Right now, adding a third backend means manually creating another instance group and VM. A managed instance group with autoscaling would make that a policy, not a manual step.
• Cloud CDN in front of the static site. The backend bucket supports it with one flag. I left it off deliberately, since it’s a clean answer to “what’s next” rather than something to enable before there’s any traffic to justify it.
• Real monitoring and alerting, beyond Cloud NAT’s connection logging — uptime checks on both load balancers and alerting policies that fire before a customer notices, not after.
• Multi-region, not just multi-zone, for the backend. Two zones protect against a zone-level outage. A regional outage would still take the whole thing down.

Commands Used to Build FreshCart Infra

I put all the commands and the order they were ran in a script, although I didnt. Here’s all the commands I used:

#!/usr/bin/env bash

# FreshCart — Two-Tier GCP Architecture, full build 

# The teardown script is intentionally NOT included, it's in a separate script.
#
# Fill in every variable below before running.

set -euo pipefail

# VARIABLES

PROJECT_ID="your-project-id"
REGION="us-central1"
ZONE="us-central1-a"          
ZONE_B="us-central1-b"        
BILLING_ACCOUNT_ID="XXXXXX-XXXXXX-XXXXXX"

DUCKDNS_SUBDOMAIN="yourname"          
DUCKDNS_TOKEN="yourname-duckdns-token"
DOMAIN="${DUCKDNS_SUBDOMAIN}.duckdns.org"

BUCKET_NAME="freshcart-static-${PROJECT_ID}"
HTML_FILE="$HOME/Downloads/freshcart-coming-soon.html"   

echo "== FreshCart build starting: $PROJECT_ID =="

#PROJECT SETUP

gcloud config set project "$PROJECT_ID"
gcloud config set compute/region "$REGION"
gcloud services enable compute.googleapis.com storage.googleapis.com

gcloud billing budgets create \
  --billing-account="$BILLING_ACCOUNT_ID" \
  --display-name="FreshCart Capstone Budget" \
  --budget-amount=10USD \
  --threshold-rule=percent=0.5 \
  --threshold-rule=percent=0.9 \
  --threshold-rule=percent=1.0 

#CREATE VPC AND TWO PRIVATE SUBNETS

gcloud compute networks create freshcart-vpc --subnet-mode=custom

gcloud compute networks subnets create freshcart-private-subnet \
  --network=freshcart-vpc \
  --region="$REGION" \
  --range=10.0.2.0/24 \
  --enable-private-ip-google-access

gcloud compute networks subnets create freshcart-private-subnet-2 \
  --network=freshcart-vpc \
  --region="$REGION" \
  --range=10.0.3.0/24 \
  --enable-private-ip-google-access

#CREATE FIREWALL RULES

gcloud compute firewall-rules create allow-lb-health-check \
  --network=freshcart-vpc \
  --direction=INGRESS \
  --action=ALLOW \
  --rules=tcp:8080 \
  --source-ranges=130.211.0.0/22,35.191.0.0/16 \
  --target-tags=backend

gcloud compute firewall-rules create allow-iap-ssh \
  --network=freshcart-vpc \
  --direction=INGRESS \
  --action=ALLOW \
  --rules=tcp:22 \
  --source-ranges=35.235.240.0/20 \
  --target-tags=backend

#CLOUD ROUTER & CLOUD NAT

gcloud compute routers create freshcart-router \
  --network=freshcart-vpc \
  --region="$REGION"

gcloud compute routers nats create freshcart-nat \
  --router=freshcart-router \
  --region="$REGION" \
  --nat-custom-subnet-ip-ranges=freshcart-private-subnet,freshcart-private-subnet-2 \
  --auto-allocate-nat-external-ips \
  --enable-logging

#CREATE TWO BACKEND VMS IN TWO SEPERATE SUBNETS

gcloud iam service-accounts create freshcart-backend-sa \
  --display-name="FreshCart backend VM"

cat > /tmp/freshcart-startup.sh << 'EOF'
#!/bin/bash
apt-get update
apt-get install -y nginx
sed -i 's/listen 80 default_server;/listen 8080 default_server;/' /etc/nginx/sites-available/default
sed -i 's/listen \[::\]:80 default_server;/listen [::]:8080 default_server;/' /etc/nginx/sites-available/default
echo "FreshCart backend is up" > /var/www/html/index.nginx-debian.html
systemctl restart nginx
systemctl enable nginx
EOF

gcloud compute instances create freshcart-backend-vm \
  --zone="$ZONE" \
  --machine-type=e2-micro \
  --image-family=debian-12 \
  --image-project=debian-cloud \
  --boot-disk-size=10GB \
  --subnet=freshcart-private-subnet \
  --no-address \
  --tags=backend \
  --service-account="freshcart-backend-sa@${PROJECT_ID}.iam.gserviceaccount.com" \
  --scopes=https://www.googleapis.com/auth/logging.write,https://www.googleapis.com/auth/monitoring.write \
  --metadata-from-file=startup-script=/tmp/freshcart-startup.sh

gcloud compute instances create freshcart-backend-vm-2 \
  --zone="$ZONE_B" \
  --machine-type=e2-micro \
  --image-family=debian-12 \
  --image-project=debian-cloud \
  --boot-disk-size=10GB \
  --subnet=freshcart-private-subnet-2 \
  --no-address \
  --tags=backend \
  --service-account="freshcart-backend-sa@${PROJECT_ID}.iam.gserviceaccount.com" \
  --scopes=https://www.googleapis.com/auth/logging.write,https://www.googleapis.com/auth/monitoring.write \
  --metadata-from-file=startup-script=/tmp/freshcart-startup.sh \
  --provisioning-model=SPOT \
  --instance-termination-action=STOP

echo "== Waiting 60s for both VMs to finish their startup script =="
sleep 60

for vm in "freshcart-backend-vm:$ZONE" "freshcart-backend-vm-2:$ZONE_B"; do
  name="${vm%%:*}"
  z="${vm##*:}"
  gcloud compute scp "$HTML_FILE" "${name}:/tmp/index.html" --zone="$z" --tunnel-through-iap
  gcloud compute ssh "$name" --zone="$z" --tunnel-through-iap --command="
    sudo mv /tmp/index.html /var/www/html/index.html &&
    sudo rm -f /var/www/html/index.nginx-debian.html &&
    sudo systemctl reload nginx
  "
done

#LOAD BALANCER AND HEALTH CHECK

gcloud compute addresses create freshcart-lb-ip --ip-version=IPV4 --global

gcloud compute health-checks create http freshcart-health-check \
  --port=8080 \
  --request-path=/ \
  --check-interval=10s \
  --timeout=5s \
  --healthy-threshold=2 \
  --unhealthy-threshold=3

gcloud compute instance-groups unmanaged create freshcart-backend-group --zone="$ZONE"
gcloud compute instance-groups unmanaged add-instances freshcart-backend-group --zone="$ZONE" --instances=freshcart-backend-vm
gcloud compute instance-groups unmanaged set-named-ports freshcart-backend-group --zone="$ZONE" --named-ports=http:8080

gcloud compute instance-groups unmanaged create freshcart-backend-group-2 --zone="$ZONE_B"
gcloud compute instance-groups unmanaged add-instances freshcart-backend-group-2 --zone="$ZONE_B" --instances=freshcart-backend-vm-2
gcloud compute instance-groups unmanaged set-named-ports freshcart-backend-group-2 --zone="$ZONE_B" --named-ports=http:8080

gcloud compute backend-services create freshcart-backend-service \
  --load-balancing-scheme=EXTERNAL_MANAGED \
  --protocol=HTTP \
  --port-name=http \
  --health-checks=freshcart-health-check \
  --global

gcloud compute backend-services add-backend freshcart-backend-service \
  --instance-group=freshcart-backend-group \
  --instance-group-zone="$ZONE" \
  --global

gcloud compute backend-services add-backend freshcart-backend-service \
  --instance-group=freshcart-backend-group-2 \
  --instance-group-zone="$ZONE_B" \
  --global

gcloud compute url-maps create freshcart-url-map \
  --default-service=freshcart-backend-service \
  --global

gcloud compute target-http-proxies create freshcart-http-proxy \
  --url-map=freshcart-url-map

gcloud compute forwarding-rules create freshcart-http-forwarding-rule \
  --load-balancing-scheme=EXTERNAL_MANAGED \
  --address=freshcart-lb-ip \
  --global \
  --target-http-proxy=freshcart-http-proxy \
  --ports=80

#GCS static site + DuckDNS + managed cert

gcloud storage buckets create "gs://$BUCKET_NAME" \
  --location="$REGION" \
  --uniform-bucket-level-access

gcloud storage buckets update "gs://$BUCKET_NAME" \
  --web-main-page-suffix=index.html \
  --web-error-page=index.html

gcloud storage cp "$HTML_FILE" "gs://$BUCKET_NAME/index.html"

gcloud storage buckets add-iam-policy-binding "gs://$BUCKET_NAME" \
  --member=allUsers \
  --role=roles/storage.objectViewer

gcloud compute addresses create freshcart-static-site-ip --ip-version=IPV4 --global

STATIC_IP=$(gcloud compute addresses describe freshcart-static-site-ip --global --format="get(address)")
echo "Static site IP: $STATIC_IP — updating DuckDNS now"
curl -s "https://www.duckdns.org/update?domains=${DUCKDNS_SUBDOMAIN}&token=${DUCKDNS_TOKEN}&ip=${STATIC_IP}"

gcloud compute backend-buckets create freshcart-static-backend \
  --gcs-bucket-name="$BUCKET_NAME"

gcloud compute ssl-certificates create freshcart-static-cert \
  --domains="$DOMAIN" \
  --global

gcloud compute url-maps create freshcart-static-url-map \
  --default-backend-bucket=freshcart-static-backend \
  --global

gcloud compute target-https-proxies create freshcart-static-https-proxy \
  --url-map=freshcart-static-url-map \
  --ssl-certificates=freshcart-static-cert

gcloud compute forwarding-rules create freshcart-static-https-rule \
  --load-balancing-scheme=EXTERNAL_MANAGED \
  --address=freshcart-static-site-ip \
  --global \
  --target-https-proxy=freshcart-static-https-proxy \
  --ports=443

#LIFECYCLE RULE

cat > /tmp/freshcart-lifecycle.json << 'EOF'
{
  "rule": [
    {
      "action": {"type": "SetStorageClass", "storageClass": "NEARLINE"},
      "condition": {"age": 30}
    },
    {
      "action": {"type": "Delete"},
      "condition": {"age": 90}
    }
  ]
}
EOF

gcloud storage buckets update "gs://$BUCKET_NAME" --lifecycle-file=/tmp/freshcart-lifecycle.json

Closing

Everything here was built and torn down through the GCP CLI— no console clicks, start to finish. That’s a genuinely different way to work than clicking through the GCP console to create and manage every resource: every decision above is legible in a command, not buried in a UI state that’s hard to reconstruct later.