GCP Reasoning Engine : Building Platform to Fetch Metrics

From Customer Ask to Cloud Monitoring API: Building Agent Metrics the Right Way on GCP

We want to show our users how their AI agents are performing — execution time, success rate,error rate — in our own portal. Here is the full journey from that requirement to a working, production-grade metrics API.

It Started with a Customer Request

We had deployed LangGraph-based AI agents to GCP Agent Engine. The agents were running. Users were querying them. Things were working.

Then came the requirement: We want to show metrics in our own portal. How long are agents taking? How many requests are succeeding? What’s the error rate?

Simple enough on the surface. But as soon as we started digging into how to actually fetch and expose this data correctly — at the right granularity, with the right numbers — a series of non-obvious decisions appeared one after another.

This blog is a walkthrough of that journey: the questions we faced, the wrong paths we considered, and the approach we settled on.

Step 1: Where Does the Data Even Come From?

The first question was: where do we get this data from?

GCP Agent Engine (built on Vertex AI Reasoning Engine) automatically emits platform-level metrics. No instrumentation required on our side. The data is already there — we just need to know where to find it and how to read it correctly.

Agent Engine emits metrics under the ‘aiplatform.googleapis.com/reasoning_engine/’ namespace.

For our three user-facing metrics, two physical GCP metric paths do all the work:

Notice that request_count powers two different derived metrics. The distinction between success rate and error rate comes entirely from a label filter — not a different metric path.

More on that shortly.

Every query scopes to a specific agent using two fields:

resource.type = "aiplatform.googleapis.com/ReasoningEngine"
resource.labels.reasoning_engine_id = "<your-agent-resource-id>"

The reasoning_engine_id is the unique ID assigned at deployment. It is the primary key for all per-agent metrics queries.

Step 2: Cloud Logging or Cloud Monitoring?

Before writing a single line of code, we had a fundamental choice to make.

Agent Engine also writes structured log entries to Cloud Logging for every request — status codes, latency, error details. It seemed possible to just parse those logs and derive the numbers ourselves. Query the logs, count the lines, calculate the averages.

We chose Cloud Monitoring instead. Here is the reasoning that settled it.

Cloud Monitoring is built for time series — Cloud Logging is not

Cloud Monitoring’s ‘ListTimeSeries’ API returns data already bucketed, aligned, and aggregated by the platform. You get pre-computed delta values, rates per second, and cumulative sums as first-class responses. With Cloud Logging you get raw structured entries — you would need to implement bucketing, alignment, gap-filling, and aggregation yourself in application code, for every query, every time.

Log ingestion lags. Platform metrics don’t.

Cloud Logging ingestion can be delayed by tens of seconds to minutes under load. Cloud Monitoring metrics are emitted by the infrastructure directly, typically available within 1–2 minutes with consistent latency. For a portal dashboard that refreshes regularly, that difference matters.

More critically: if an agent crashes before emitting a log line, that request disappears from the logs entirely. Infrastructure-level metrics from Cloud Monitoring are emitted by the platform — they are present even when the agent fails silently.

Log queries get expensive fast

Cloud Logging’s entries.list is built for log exploration, not high-frequency programmatic queries. Running it on every dashboard refresh, for every agent, against potentially millions of log entries is slow and costly.

Response code classification is already built in

This was the deciding factor. Agent Engine’s request_count metric already has a ‘response_code_class’ label that classifies every request:

Splitting success from error is one AND clause in a filter string. With log parsing you would need to extract status codes from JSON, handle missing fields, and re-implement this classification — for no accuracy gain.

The one case where logs are the right choice

Logs are irreplaceable when you need the content of what happened: what did the model respond? What was the stack trace? What custom events did the agent emit? For time series metrics — rates, averages, counts over time — Cloud Monitoring is always the better tool.

Decision: Cloud Monitoring via ‘google-cloud-monitoring’ SDK.

Step 3: Understanding the GCP APIs We’re Working With

Once we committed to Cloud Monitoring, we needed to understand the SDK surface. All queries go through MetricServiceClient from the google.cloud.monitoring_v3 package. Under the hood, the SDK calls the Cloud Monitoring API.

Here are the key endpoints and methods used:

• API Endpoint: ‘monitoring.googleapis.com’
• Method: ‘projects.timeSeries.list’

The key building blocks:

Authentication uses Application Default Credentials (ADC)— no extra credential wiring needed in GKE or Cloud Run. The required IAM role is roles/monitoring.viewer. The most important concept to internalise: the aligner determines what number you get.

The same raw request_count metric produces completely different results depending on whether you use ALIGN_RATE (queries per second) or ALIGN_SUM (total count). Getting the aligner wrong means getting silently wrong answers.

Step 4: Designing the Query Structure

With the tools understood, we designed a shared query structure that all three metrics follow. Every request from the portal specifies:

• resource_id — which agent
• metric_type — which derived metric
• start_time / end_time — the time window
• bucket_size — how to group time (5m, 1h, 6h, 1d, or auto)

Making bucket size automatic

A portal user should not need to think about alignment periods. When auto is selected (the default), the bucket size adapts to the window length:

A 1-hour window gets 5-minute buckets — 12 data points, readable on a chart. A 7-day window gets 6-hour buckets — 28 data points, enough granularity without overwhelming the chart. The caller never needs to reason about Cloud Monitoring alignment periods.

The response envelope

Every metric returns the same structure, regardless of type:

{
"resource_id": "…",
"metric_type": "success_request_rate",
"unit": "queries/sec",
"points": [
{ "timestamp": "2026–04–15T09:00:00Z", "value": 0.013 },
{ "timestamp": "2026–04–15T10:00:00Z", "value": 0.018 }
],
"summary": { … }
}

• points — the time series array, ready to drop into a chart component
• summary — pre-computed headline numbers for KPI cards (totals, percentages, min/max)
• unit — the unit of the points values

This separation means the portal can render a chart from points and a KPI card from summary with a single API call, no post-processing needed on the front end.

Step 5: Solving Each Metric — The Decisions That Weren’t Obvious

average_execution_time — the zero-bucket trap

The naive approach: query request_latencies with ALIGN_DELTA, average all the bucket values, done.

The problem: Cloud Monitoring gap-fills time series. If the agent had no traffic during a time bucket, it still emits a data point with value 0. Averaging those zeros into the result makes an agent that ran once look like it has near-zero execution time.

The fix: exclude zero-value buckets before averaging. Only buckets with actual executions contribute to the average.

active_buckets = [v for v in all_bucket_values if v > 0]
average = sum(active_buckets) / len(active_buckets)

This was not an edge case — in any realistic deployment, agents have idle periods.

Without this filter, every metric would be wrong by design.

Why ALIGN_DELTA and not just ALIGN_MEAN?

ALIGN_MEAN would give the within-bucket mean latency, but Cloud Monitoring’s request_latencies is a distribution metric. ALIGN_DELTA on a distribution gives the total latency mass across the bucket, which we then average across active buckets. It produces a more stable and intuitive “average time per run” for the window.

last_execution_at — why it needs its own query

The portal also needed to show “last active” — when did this agent last receive a request? This cannot be derived from an averaged latency series.

The approach: run a separate targeted query on request_latencies with a fixed 1-minute alignment period, independent of the user’s chosen bucket size. Scan the results from start_time to now. The timestamp of the last non-zero bucket is last_execution_at.

Why fix the alignment at 1 minute? If the alignment period matched the user’s bucket size, the timestamp would shift by up to one full bucket length (potentially 6 hours) between API calls. A fixed 1-minute alignment gives ±1 minute accuracy and a stable, consistent result on every call.

success_request_rate — rate vs count requires two queries

The portal needs two different things from the same metric:

• A rate (queries/sec) for the chart — normalised for bucket size comparison
• An absolute count for the KPI card — “1,132 successful requests in the last 24 hours”
• A percentage — “92.2% of all requests succeeded”

These cannot come from a single query. ALIGN_RATE produces the per-second rate.

ALIGN_SUM produces the raw count. So we run two queries:

1. ALIGN_RATE → feeds points[] for the chart
2. ALIGN_SUM → feeds success_request_count in the summary

For the percentage, a third query fetches unfiltered request_count (no

response_code_class filter) with ALIGN_SUM. That total becomes the denominator:

success_rate_percentage = (success_count / total_count) × 100

Three queries for one metric sounds expensive but they are all lightweight Cloud Monitoring time series queries — each returns at most a few dozen data points. The latency cost is negligible compared to the correctness gain.

error_request_rate — the 4xx/5xx decision

The filter for error rate is:

metric.labels.response_code_class = "4xx" OR 
metric.labels.response_code_class = "5xx"

Both client errors (4xx) and server errors (5xx) are counted as errors. The reasoning:

from the portal user’s perspective, any failed request is a failure — regardless of whether the agent caused it or the caller sent bad input.

• If you want to separate** agent reliability (5xx) from caller errors (4xx), run two separate queries with distinct filters and expose them as server_error_rate and client_error_rate. The mechanism is identical — only the filter value changes.

Step 6: The Full Query Flow

Putting it all together, here is what happens on every API call:

What We Learned

Looking back at the journey from customer ask to working API, the non-obvious lessons were:

1. The right data source is Cloud Monitoring, not Cloud Logging. Platform-emitted metrics are pre-aligned, gap-filled, and always available — even when agents crash without logging.
2. Two GCP metric paths power all three derived metrics. request_latencies and request_count — with different aligners and response_code_class filters.
3. The aligner is the most important parameter. ALIGN_RATE, ALIGN_DELTA, and ALIGN_SUM on the same metric path produce entirely different numbers. Get it wrong and the API returns silently incorrect data.
4. Always filter out zero-value buckets before averaging. Cloud Monitoring gap-fills idle periods with zero. Ignoring this makes every average wrong.
5. Rate and count require two separate queries. There is no single aligner that produces both a per-second rate and an absolute count.
6. last_execution_at needs a fixed alignment period, separate from the user’s bucket size. Without this, the timestamp drifts by seconds on every API call.
7. Return points (chart) and summary (KPIs) in a single response. The portal should not need two API calls to render one dashboard card.
8. Percentages must use total unfiltered request count as the denominator. Success % + Error % ≤ 100% holds mathematically and accounts for any unmapped status codes.

References

For further information, refer to the following Google Cloud documentation:

• Cloud Monitoring Python Client Library: Python Client for Cloud Monitoring
• Cloud Monitoring API Reference: projects.timeSeries.list
• Google Cloud Metrics: Metric types (Look for aiplatform.googleapis.com/reasoning_engine/)
• Vertex AI Reasoning Engine: Overview