Deploying gRPC and REST Services Across Multi-Project Kubernetes Clusters on GCP

In modern cloud-native architectures, deploying services across multiple Google Cloud (GCP) projects using Kubernetes clusters, gRPC, and REST APIs requires careful planning — especially when it comes to VPC networking, DNS resolution, and load balancing. This post outlines a comprehensive, production-grade deployment of a backend service in GKE (Google Kubernetes Engine) that exposes both gRPC and REST APIs. The approach includes VPC peering, DNS configuration, load balancer setup, and ingress management.

🌟 Objectives

• Deploy a backend service exposing gRPC (port 50051) and REST (port 8080) endpoints.
• Enable secure public access and private internal access (via VPC peering).
• Utilize DNS-based service discovery across projects.
• Implement internal and external load balancing.
• Ensure observability and proper health checks.

🛡️ VPC Peering Between Multi-Project Clusters

Setup in Project A (e.g., platform-dev)

1. Navigate to VPC network > VPC network peering.
2. Create a new peering connection:

• Select local VPC.
• For peer project ID, enter Project B’s ID.
• For peer VPC name, use the VPC network from Project B (e.g., trace-net).

3. Save and verify status becomes “Active”.

Get Source Ranges from Project B

CIDR Range Explanation

CIDR (Classless Inter-Domain Routing) notation defines IP address blocks. The number after the slash (e.g., /24, /17) indicates how many bits are fixed in the network prefix.

• /24 means 256 IP addresses (e.g., 10.0.0.0 to 10.0.0.255).
• /17 means 32,768 IP addresses (e.g., 10.13.0.0 to 10.13.127.255).

A smaller number (like /17) means a larger IP range. Use a larger range when your Kubernetes cluster has many pods and services needing unique IPs. To configure firewalls, you need the primary and secondary IP ranges of the peer project:

gcloud config set project <peer-project-id>
gcloud compute networks subnets list \
  --filter="region:asia-southeast1 AND network:trace-net" \
  --format="table(name, network, ipCidrRange, secondaryIpRanges)"

Look for CIDR ranges such as:

• 10.0.0.0/24
• 10.2.0.0/24
• 10.13.0.0/17

Configure Firewall Rules

Use the gathered ranges to allow inter-project communication:

gcloud compute firewall-rules create allow-interproject \
  --network trace-net \
  --allow tcp:8080,tcp:50051 \
  --source-ranges=10.0.0.0/24,10.2.0.0/24,10.13.0.0/17

🤜 Internal Load Balancer Setup (GKE)

An Internal Load Balancer (ILB) in GCP is used for routing traffic within a VPC or across peered VPCs, making it ideal for services that should not be exposed to the public internet. These are only accessible from within the internal network, offering greater security for backend services.

In contrast, an External Load Balancer exposes services to the internet, allowing public access to applications. This type of load balancer typically includes URL-based routing and supports managed certificates for secure HTTPS communication.

Below is the configuration for setting up an internal load balancer to expose gRPC and REST endpoints internally:

apiVersion: v1
kind: Service
metadata:
  name: internal-api-service
  namespace: default
  annotations:
    cloud.google.com/load-balancer-type: "Internal"
    cloud.google.com/app-protocols: '{"grpc":"HTTP2", "rest":"HTTP"}'
    cloud.google.com/backend-config: '{"default": "k8-timeout-config"}'
spec:
  type: LoadBalancer
  selector:
    app: api-backend
  ports:
    - name: rest
      port: 8080
      targetPort: 8080
    - name: grpc
      port: 443
      targetPort: 50051

Backend config

apiVersion: cloud.google.com/v1
kind: BackendConfig
metadata:
  name: k8-timeout-config
spec:
  timeoutSec: 60
  logging:
    enable: true
    sampleRate: 1
  healthCheck:
    checkIntervalSec: 10
    timeoutSec: 5
    healthyThreshold: 2
    unhealthyThreshold: 2
    type: HTTP
    requestPath: /health/liveness
    port: 8080

Apply all components:

kubectl apply -f backendconfig.yaml
kubectl apply -f internal-api-service.yaml

Retrieve the Internal IP Address

Use this command to get the internal IP address assigned by the ILB:

kubectl get svc internal-api-service

Check the EXTERNAL-IP column, which will actually reflect the internal IP (e.g., 10.2.0.50). This IP is used for private DNS resolution and internal access.

🌐 DNS Setup for Cross-Project Service Discovery

In Project B:

1. Create a Private DNS Zone named internal.services.
2. Add A records such as:

• api-service.internal.services. -> 10.2.0.50

3. Link the zone to the VPC (e.g., trace-net).

In Project A:

1. Create a Private DNS Zone with the same DNS name: internal.services.
2. Set up DNS Peering:

• Peer Project ID: Project B
• Peer Network: trace-net

Verification:

Step 1: Verify DNS Resolution

Run this command inside a pod in Project A:

kubectl exec -it <pod-name> -- getent hosts api-service.internal.services.

Expected output:

10.2.0.50   api-service.internal.services.

Step 2: Verify Service Reachability

Use curl to hit the service endpoint and check readiness:

kubectl exec -it <pod-name> -- curl http://api-service.internal.services.:8080/health/readiness

Expected output:

{"status":"UP"}

Step 3: Pod-to-Pod Internal Communication (Optional)

Get the internal IP of a pod in Project A and test reachability from Project B:

kubectl get pod <pod-name> -o wide
kubectl exec -it <other-pod-in-project-b> -- curl http://<internal-ip>:8080/healthz

✅ Summary

This approach enables a scalable, secure, and modular deployment of a dual-mode (gRPC and REST) service architecture in GCP. It ensures both public and private traffic routing is optimized through proper networking, DNS configuration, and load balancing.

For reusable YAML templates or further guidance, feel free to reach out in the comments!