Debugging Performance Issues Across

On-Prem and Cloud-Native Microservices

Introduction

Most performance write-ups either stay too abstract (“check your connection pool”) or too narrow (one stack trace, no context). This post does neither. It walks through two concrete architectures — a classic on-prem Dockerized microservice stack, and a GCP cloud-native stack — and traces a real-world class of incident end-to-end: symptom, root cause, debug commands, fix, and prevention.

If you operate Spring Boot / Spring Batch services backed by PostgreSQL, or you’re running the equivalent on Cloud Run with Firestore/Cloud SQL, this is written for you.

Part 1 — On-Prem Stack: Angular → API Gateway → Spring Batch → PostgreSQL (Docker, Linux)

The Architecture

A typical mid-size setup: an Angular SPA talking to an API Gateway, which fronts a set of Spring Boot microservices, one of which runs a nightly Spring Batch reconciliation job — all against a single shared PostgreSQL instance, all on Docker on a Linux host.

On-prem microservice stack — Angular UI, API Gateway, Spring Boot services, and Spring Batch sharing one PostgreSQL instance on Docker.

This is a completely reasonable architecture for a v1–v2 product. The problem isn’t the architecture itself — it’s what happens when batch and interactive workloads are left to share infrastructure without isolation.

The Incident (Real-World Pattern)

Symptom reported by users (09:00–09:15): The Angular dashboard hangs on load. Some requests return after 25–30 seconds, many return HTTP 504, and a growing number of users start hammering refresh, making things worse.

What was actually happening underneath:

Failure cascade — a long-running batch job exhausts the shared connection pool, starving interactive traffic and triggering a retry storm.

1. The Spring Batch reconciliation job kicked off at 02:00 and was still running at 09:00 because the dataset had grown 3x since the job was last tuned.

2. The batch job’s chunk size (500 records/chunk) combined with row-level UPDATE statements caused it to hold transactions open far longer than expected, consuming 18 of the 20 connections in the shared HikariCP pool.

3. At 09:00, normal business-hours traffic started hitting the Order Service, which needed connections from the same pool.

4. Requests queued on HikariPool-1 waiting for a connection, hit the default connectionTimeout of 30 seconds, and failed with SQLTransientConnectionException.

5. The API Gateway’s own request queue backed up behind these slow/failing calls, which made unrelated endpoints slow too — the gateway’s thread pool was the second-order victim.

6. Users retried, multiplying load on an already-starved pool. Classic retry storm.

Debug Steps (In the Order We Actually Used Them)

Step 1 — Confirm it’s a DB/connection issue, not a CPU/network issue

# Container-level resource check first — rule out the obvious

docker stats — no-stream

# Check Spring Boot Actuator health + pool metrics (if exposed)

curl -s http://localhost:8080/actuator/health | jq

curl -s http://localhost:8080/actuator/metrics/hikaricp.connections.active | jq

curl -s http://localhost:8080/actuator/metrics/hikaricp.connections.pending | jq

A high hikaricp.connections.pending value alongside hikaricp.connections.active near maximum-pool-size is the smoking gun.

Step 2 — Check application logs for the exact exception

docker logs order-service — since 30m | grep -i “SQLTransientConnectionException|HikariPool”

Typical output:

2026–06–20 09:04:12.331 WARN [http-nio-8080-exec-7] com.zaxxer.hikari.pool.HikariPool

HikariPool-1 — Connection is not available, request timed out after 30001ms.

Step 3 — Confirm what’s holding connections on the PostgreSQL side

— Run inside the Postgres container

SELECT pid, usename, application_name, state, wait_event_type, wait_event,

now() — query_start AS running_for, query

FROM pg_stat_activity

WHERE state != ‘idle’

ORDER BY running_for DESC;

This is where you’ll see the batch job’s connections sitting at the top, often with running_for in the 5–10+ minute range and wait_event_type = Lock.

Step 4 — Check for lock contention specifically

SELECT blocked_locks.pid AS blocked_pid,

blocked_activity.query AS blocked_query,

blocking_locks.pid AS blocking_pid,

blocking_activity.query AS blocking_query

FROM pg_catalog.pg_locks blocked_locks

JOIN pg_catalog.pg_stat_activity blocked_activity ON blocked_activity.pid = blocked_locks.pid

JOIN pg_catalog.pg_locks blocking_locks

ON blocking_locks.locktype = blocked_locks.locktype

AND blocking_locks.pid != blocked_locks.pid

JOIN pg_catalog.pg_stat_activity blocking_activity ON blocking_activity.pid = blocking_locks.pid

WHERE NOT blocked_locks.granted;

This confirms whether batch UPDATEs are row/table-locking against rows the Order Service also needs to read or write.

Step 5 — Correlate with Spring Batch job execution metadata

SELECT job_execution_id, job_name, start_time, end_time, status

FROM batch_job_execution

ORDER BY start_time DESC

LIMIT 5;

If end_time is NULL and start_time is hours ago, the job is still running well past its expected window — your actual root cause.

Root Cause Summary

Layer — What Went Wrong

Spring Batch

Chunk size too large for current data volume; job runtime grew silently as data grew

Connection pool

Shared HikariCP pool between batch and OLTP traffic, no isolation

PostgreSQL

No statement timeout, no lock timeout — long transactions allowed to run unbounded

API Gateway

No circuit breaker/bulkhead between this service and others, so the failure spread

Monitoring

No alert on hikaricp.connections.pending or batch job duration — issue was discovered by users, not by alerting

Fix — Immediate (incident mitigation)

— Kill the offending long-running batch query if it’s safe to do so

SELECT pg_terminate_backend(<pid>);

Then restart the batch job during a quieter window, or let it resume from its last committed chunk (Spring Batch supports restart-from-failure if configured with a JobRepository).

Fix — Short-term

• Give the batch job its own dedicated HikariCP pool (separate DataSource bean), sized independently from the OLTP pool.

• Set connectionTimeout sensibly (e.g., 5s for OLTP, can stay higher for batch) so OLTP fails fast and visibly rather than hanging for 30s.

• Add statement_timeout and lock_timeout at the Postgres role level for the batch user:

ALTER ROLE batch_user SET statement_timeout = ‘300s’;

ALTER ROLE batch_user SET lock_timeout = ‘10s’;

Spring Boot Config Example — Separate Pools

spring:

datasource:

oltp:

jdbc-url: jdbc:postgresql://db:5432/appdb

hikari:

maximum-pool-size: 20

connection-timeout: 5000

pool-name: OltpPool

batch:

jdbc-url: jdbc:postgresql://db:5432/appdb

hikari:

maximum-pool-size: 10

connection-timeout: 30000

pool-name: BatchPool

Tune the Batch Job

@Bean

public Step reconciliationStep(JobRepository jobRepository,

PlatformTransactionManager transactionManager) {

return new StepBuilder(“reconciliationStep”, jobRepository)

.<Order, Order>chunk(100, transactionManager) // reduced from 500

.reader(orderReader())

.processor(orderProcessor())

.writer(orderWriter())

.faultTolerant()

.skipLimit(50)

.build();

}

Smaller chunk size = shorter-held transactions = less lock contention, at the cost of slightly more overhead per chunk. Measure and tune for your data shape.

Prevention Checklist (On-Prem)

☐ Isolate pools — batch and interactive traffic never share a DataSource.

☐ Set timeouts everywhere — connection timeout, statement timeout, lock timeout. Silent infinite waits are the enemy.

☐ Alert on pool saturation — alert when hikaricp.connections.pending > 0 for more than N seconds, not just on CPU/memory.

☐ Alert on batch job duration drift — if a job historically runs 20 minutes and is now running 90, that’s a leading indicator, not a trailing one.

☐ Bulkhead at the gateway — circuit breakers (Resilience4j) so one service’s DB issue doesn’t cascade through the gateway to unrelated services.

☐ Load test with batch + OLTP running concurrently — most teams only load-test one workload at a time and miss this entirely.

☐ Right-size max_connections vs application pool sizing — total of all pools across all service instances should have real headroom under Postgres’s max_connections, accounting for admin/replication connections too.

Part 2 — GCP Cloud-Native Stack: Cloud Run, Java 21, Datastore, Spring Boot

The Architecture

Same general shape, rebuilt cloud-native: Cloud Load Balancer → API Gateway → Cloud Run (Spring Boot on Java 21) → Firestore in Datastore mode (or Cloud SQL for relational needs).

GCP cloud-native stack — Cloud Load Balancer, API Gateway, autoscaled Cloud Run instances, and Firestore/Cloud SQL.

This architecture removes some on-prem problems (no more single Docker host, autoscaling is automatic) but introduces new failure modes that don’t exist on-prem.

Where Performance Issues Actually Come from Here

1. Cold Starts

Cloud Run scales to zero by default. A Java 21 Spring Boot app has real JVM startup cost — even with Spring Boot 3’s improvements and AOT/CDS support, a cold instance can add 1–3+ seconds to the first request after a scale-up event.

Debug:

gcloud run services describe order-service — region=us-central1 \

— format=”value(status.traffic)”

# Pull recent request logs and look for high latency on first requests after idle gaps

gcloud logging read ‘

resource.type=”cloud_run_revision”

resource.labels.service_name=”order-service”

httpRequest.latency>”2s”

‘ — limit=50 — format=json

Fix:

• Set — min-instances=1 (or more) for latency-sensitive services to avoid scale-to-zero entirely.

• Use Spring Boot’s Class Data Sharing (CDS) with Java 21 and a layered/AOT build to cut JVM startup time.

• Use — cpu-boost on Cloud Run to allocate extra CPU during startup.

2. Per-Instance Connection Pooling vs. a Finite DB Connection Budget

This is the cloud-native cousin of Part 1’s problem, but worse, because autoscaling multiplies it automatically. If each Cloud Run instance opens a HikariCP pool of 10 connections, and the service scales to 30 instances under load, that’s 300 potential connections against a Cloud SQL instance that might cap at max_connections = 100.

Debug:

# Check current Cloud SQL connection count

gcloud sql operations list — instance=prod-db — limit=5

# Better — query directly

gcloud sql connect prod-db — user=postgres

# then inside psql:

SELECT count() FROM pg_stat_activity;*

SHOW max_connections;

# Check how many Cloud Run instances are currently active

gcloud run services describe order-service — region=us-central1 \

— format=”value(status.observedGeneration, status.traffic[0].percent)”

gcloud monitoring time-series list \

— filter=’metric.type=”run.googleapis.com/container/instance_count”

Fix:

• Route through the Cloud SQL Auth Proxy (or the built-in Cloud SQL connector for Cloud Run) and set a hard per-instance pool ceiling that accounts for max scale: max_instances × maximum-pool-size ≤ Cloud SQL max_connections × safety_margin.

• Set — max-instances explicitly on Cloud Run rather than leaving it unbounded.

• Consider Cloud SQL connection pooling via PgBouncer as a shared intermediary so per-instance pools don’t directly multiply against the DB.

spring:

datasource:

hikari:

maximum-pool-size: 5 # deliberately small — multiply by max-instances

minimum-idle: 1

connection-timeout: 3000

gcloud run deploy order-service \

— max-instances=15 \

— concurrency=80 \

— cpu-boost \

— min-instances=1

5 connections × 15 max instances = 75, safely under a 100-connection Cloud SQL instance with headroom for admin connections.

3. Datastore/Firestore-Specific Limits

If you’re using Datastore mode instead of Cloud SQL, the failure mode is different — it’s not connection exhaustion, it’s hot key contention and quota throttling.

Debug:

gcloud logging read ‘

resource.type=”datastore_database”

severity>=WARNING

‘ — limit=50

# Look specifically for contention errors

gcloud logging read ‘

jsonPayload.message=~”ABORTED|contention”

‘ — limit=50

Common symptom: writes to entities sharing the same key prefix (e.g., a counter entity, or sequential IDs) throttle because Datastore’s underlying storage shards by key range, and hot, sequential keys land on the same shard.

Fix:

• Avoid monotonically increasing keys for high-write entities; use sharded counters or randomized/hashed key prefixes.

• Batch writes where possible instead of many small individual writes.

• Use exponential backoff with jitter on retries (Spring Cloud GCP’s Datastore client does this, but confirm your retry policy isn’t naive fixed-delay).

4. API Gateway / Load Balancer Level Latency

# Check Cloud Load Balancer backend latency directly

gcloud monitoring time-series list \

— filter=’metric.type=”loadbalancing.googleapis.com/https/backend_latencies”’

# Check API Gateway specific logs for 5xx clustering

gcloud logging read ‘

resource.type=”api_gateway”

httpRequest.status>=500

‘ — limit=50

If backend latency is high but Cloud Run instance CPU/memory looks fine, the bottleneck is almost always downstream (DB, Datastore quota, or a slow external dependency) — not the gateway or LB itself. Don’t waste time tuning LB settings when the LB is just faithfully reporting a slow backend.

Prevention Checklist (GCP Cloud-Native)

☐ Set — min-instances ≥ 1 for any latency-sensitive service — don’t pay the cold start tax on user-facing paths.

☐ Cap — max-instances deliberately and do the math: max_instances × pool_size ≤ DB max_connections.

☐ Use Cloud SQL Auth Proxy / built-in connector, not raw IP connections, for reliability and connection management.

☐ Avoid hot keys in Datastore — design key schemas for write distribution, not just logical grouping.

☐ Alert on Cloud Run instance count + DB connection count together — a spike in one without headroom in the other is your early warning.

☐ Use Cloud Monitoring SLO alerting on backend latency percentiles (p95/p99), not just averages — averages hide exactly the kind of tail latency this incident produces.

☐ Load test with realistic scale-out — testing against a single warm instance won’t reveal connection multiplication issues.

Part 3 — Remediation & Prevention: What Actually Generalizes

Both incidents above — on-prem and cloud-native — trace back to the same handful of root causes wearing different clothes. This is the part worth pinning to your team wiki.

The Four Recurring Root Causes

1. Shared, unbounded resource pools (DB connections, threads) used by workloads with very different usage patterns (batch vs. interactive).

2. Missing timeouts at every layer — connection, statement, lock, HTTP client — meaning failures degrade slowly and invisibly instead of failing fast and loudly.

3. No isolation between workloads — one slow consumer of a shared resource can starve everyone else, and failures cascade upward through gateways into unrelated traffic.

4. Monitoring tuned for infrastructure health, not resource saturation — CPU/memory looked fine in both incidents. The actual leading indicators (pending connections, job duration drift, lock wait time) weren’t being watched.

A Real Example, Fully Worked: Connection Pool Sizing Math

This is the calculation most teams skip and then get bitten by. Here’s how to actually do it.

Inputs you need:

• DB max_connections (Postgres default is 100; Cloud SQL varies by tier)

• Reserve some headroom for admin/replication/migration connections — typically 10–20%

• Number of service instances at peak scale (fixed for on-prem, max-instances for Cloud Run)

• Number of distinct pools per instance (OLTP pool, batch pool, any reporting pool, etc.)

Formula:

usable_connections = max_connections × (1 — reserve_fraction)

per_instance_budget = usable_connections / max_instances

per_pool_size = per_instance_budget / number_of_pools_per_instance

Worked example (matches the GCP incident above):

max_connections = 100

reserve_fraction = 0.15 → usable = 85

max_instances = 15

per_instance_budget = 85 / 15 ≈ 5.6 → round down to 5

number_of_pools = 1 (assume single pool per instance here)

per_pool_size = 5

This is exactly why maximum-pool-size: 5 was the right number above, not an arbitrary guess. Do this math explicitly for your own numbers — don’t copy the constant.

Prevention Framework to Apply to Any New Service

Closing Thought

Both architectures in this post are sound on paper. Neither incident was caused by a bad technology choice — Spring Batch, PostgreSQL, Cloud Run, and Datastore are all reasonable, mature choices. The failures came from unexamined assumptions about resource sharing under concurrent load, and they were invisible to standard infrastructure monitoring until users felt them directly.

The fix, in both cases, wasn’t a rewrite. It was making implicit resource limits explicit — pool sizes, timeouts, isolation boundaries — and then watching the right saturation metrics so the next version of this incident gets caught at 2 AM by an alert, not at 9 AM by your users.

Have a related incident or a different failure mode you’ve debugged in contact-center integration, Spring, or GCP environments? That’s exactly the kind of detail that makes posts like this useful for other engineers reading it.