Building Streaming Pipelines on GCP: A Field Guide to What Goes Wrong

“What I cannot create, I do not understand.” — Feynman

I spent three months building a real-time order processing pipeline on GCP and I was genuinely proud of it. Pub/Sub feeding Dataflow, Dataflow writing to BigQuery, dashboards updating live. Green job graphs. Everything processing under a second. The kind of thing you screenshot and put in a design doc. Then production happened.

This is a write-up of what actually broke, why, and what I’d do differently. Some of it is embarrassing. All of it is real.

Why streaming at all

The requirement was fraud detection at checkout. If you batch that — even with a 5-minute window — you’re approving fraudulent orders and shipping product before you even know something’s wrong. We needed a fraud signal in BigQuery within 500ms of an order being placed. Batch processing couldn’t meet that. Streaming it is.

The stack: Cloud Pub/Sub to receive incoming order events, Dataflow (Apache Beam, Python SDK) to process them, and BigQuery via the Storage Write API to store the results. Along the way, the pipeline makes external calls to a fraud scoring API, an inventory service, a customer tier lookup, and a Vertex AI model that predicts purchase intent.

On paper this is pretty standard. In practice I made almost every mistake available to me.

The sequential pipeline

The first version of the pipeline was embarrassingly naive in retrospect. Each enrichment step ran after the previous one finished:

Order event
  → Fraud API (~300ms)
  → Customer tier lookup (~80ms)
  → Inventory check (~120ms)
  → Vertex AI intent (~200ms)
  → BigQuery write

Total time per order: 700–800ms. Already over budget, and I hadn’t even added error handling yet.

The fix is obvious once you see it: these four calls don’t depend on each other at all. You don’t need the fraud score before you check inventory. So I fired all four at the same time using asyncio.gather — wait for the slowest one, then merge the results and write.

Order event
  ↓
  ├── Fraud API
  ├── Customer tier
  ├── Inventory
  └── Vertex AI intent
       ↓ (await all)
  Merge → BigQuery

Median latency dropped from ~760ms to ~340ms. That’s a 2.2x improvement — just from not waiting unnecessarily. I suspect most pipelines making external calls have this problem and nobody measures it carefully enough to notice.

The __init__ vs setup() bug

This one wasted two days and produced the most confusing error messages I’ve seen in a while.

I’d initialized the HTTP client for the fraud API in __init__:

class FraudEnrichDoFn(beam.DoFn):
    def __init__(self):
        self.client = httpx.AsyncClient()  # wrong

Here’s what happens: Dataflow packages up your DoFn objects and ships them to remote workers using Python’s pickle serialization. The problem is that an HTTP client holds open network connections — and open connections can’t be packaged and shipped this way. So the worker unpacks a DoFn with a broken client, then fails with a cryptic connection error the first time it tries to use it. Not on startup. Not in an obvious way. Just a confusing crash mid-processing.

The correct pattern is setup():

class FraudEnrichDoFn(beam.DoFn):
    def setup(self):
        self.client = httpx.AsyncClient()  # correct — runs post-deserialization, once per worker

    def teardown(self):
        asyncio.get_event_loop().run_until_complete(self.client.aclose())

setup() runs once per worker after it's been unpacked on the remote machine. It's the right place for anything that holds a live connection — HTTP clients, gRPC channels, database connection pools. __init__ is only for simple config values that can be safely packaged. The Beam docs say this clearly; I didn't read them carefully enough.

Vertex AI cold starts and cascade failures

The Vertex AI endpoint was set to scale down to zero when idle — no traffic, no running containers. Reasonable for saving cost. Catastrophic in a streaming context.

After a quiet period — say, early morning — the first order hit the endpoint and waited 4–6 seconds for a container to spin back up. That’s long enough for Dataflow to give up on the request and retry it. The retry also hit an endpoint that was still warming up. Now I had more and more retries piling up. The number of unprocessed messages in Pub/Sub went from near-zero to ~40,000 in about eight minutes.

The fix: set min-replica-count=1 so at least one container is always running. Yes, it costs money to keep something alive doing nothing — roughly $50–70/month for a small endpoint. But the alternative is a failure spiral that wakes you up at 2am and takes 20 minutes to clear out. Easy tradeoff in hindsight, obvious only after you've been paged.

The deeper point: if your pipeline’s worst-case latency budget is 500ms but one of your dependencies takes 5 seconds to wake up, you don’t actually have a 500ms pipeline. You have one that works 99% of the time and falls apart the other 1% — in a way that makes the 99% worse too, because the backlog grows.

The silent failure problem — the worst one

This is the part that genuinely bothered me.

The fraud API had an error rate. Not high — maybe 0.3% of calls came back with a server error. My error handling at the time was:

try:
    fraud_result = await self.client.post(...)
    fraud_score = fraud_result.json()["score"]
except Exception:
    fraud_score = None  # just continue

So when the API failed, the row still got written to BigQuery — just with fraud_score = NULL. No exception raised, no dead-letter queue, no alert fired. Dataflow showed green. BigQuery showed rows. Everything looked fine.

Three weeks later, an analyst noticed that fraud score coverage had quietly dropped from ~99.8% to ~96% over the past month. The fraud API had increased its error rate during one of their deployments. We’d been writing ~150 orders per day with no fraud signal for weeks, and the pipeline had been happily treating this as success.

Null data is worse than missing data, because missing data is visible. Null data looks like presence.

What I should have done — and what I did after — is decide upfront what happens when each external call fails:

Fraud API failure:
  - retry 3x with exponential backoff
  - on final failure: route to dead-letter topic, do NOT write partial row
  - alert if DLQ rate > 0.1% over 5min window

What to do on failure is a product decision, not something you leave to a default. “Skip and continue” is a valid choice — but it needs to be a deliberate one, written down, and monitored. except Exception: pass is not a failure strategy.

Train-serve skew in the ML step

The Vertex AI model classified purchase intent. It was trained on raw product name strings from the order database:

"nike air men running wide"
"sony wh1000xm5 blk"

At some point, someone upstream added a step to clean up product names before they hit the pipeline:

"Men's Running Shoe – Wide Fit (Nike Air)"
"Sony WH-1000XM5 Wireless Headphones (Black)"

The model had never seen this cleaner format during training. Different capitalization, different structure, different vocabulary. Accuracy dropped noticeably — I’d estimate 12–15% worse on the intent classification task, though I don’t have the exact number anymore.

This is called train-serve skew: the data your model sees in production looks different from the data it was trained on. It’s silent, it’s slow, and it’s easy to miss.

The fix is boring: the cleanup logic should have been versioned and shared between whoever managed the data pipeline and whoever owned the training pipeline. There should have been a written contract saying “the model expects input in format X.” We had neither. Just a verbal agreement between two engineers who both assumed the other was handling it.

Storage Write API and the illusion of exactly-once

The Storage Write API in COMMITTED mode prevents duplicate rows in BigQuery — it deduplicates writes using stream offsets. That’s a genuinely useful guarantee.

What it doesn’t cover is everything that happened before the write. If a worker crashes after calling the fraud API but before finishing the BigQuery write, Beam retries the whole bundle. The fraud API gets called again. The inventory service gets called again. If those services log the request, update a counter, or write to their own tables — that happens twice.

This isn’t a flaw in the Storage Write API. It’s just how streaming systems work at the processing layer: messages might be processed more than once. So every operation your pipeline performs needs to be safe to repeat — meaning if you run it twice with the same input, the result is the same as running it once. We handled this by adding idempotency keys to our fraud API requests, so their side effects were safe to replay. Most serious APIs will support this if you ask.

Synthetic monitoring — the thing I wish I’d built first

Every five minutes, we inject a fake order with known values into the pipeline — a synthetic event where we already know exactly what the output should be. A separate Cloud Function checks BigQuery five minutes later, finds that fake order’s row, and validates the fraud score, intent classification, and tier enrichment against the expected values. If anything’s off, it pages.

Dataflow’s job health metrics tell you whether the job is running. They don’t tell you whether it’s producing correct output. For three weeks, I confused these two things.

Green job status = job is alive. That’s it. Whether the data is actually correct is a separate question you have to answer yourself.

Reflections

The actual Beam code for this pipeline is maybe 400 lines. The hard stuff — deciding what happens on failure, keeping latency budgets honest, managing ML data contracts, building monitoring that catches correctness problems and not just crashes — took far more thought than the code itself.

I suspect this is true for most streaming systems. Writing a DoFn is not the hard part. The hard part is being precise about what “working” actually means. A pipeline that runs without throwing exceptions is not the same as a pipeline that produces correct data. That distinction, obvious in retrospect, was not obvious to me when I was staring at a green Dataflow graph and feeling good about myself.

If I were starting over, I’d write the canary monitor first — before any real data flows through. Force yourself to define “correct output” in code before you build the thing that’s supposed to produce it. Everything else gets easier once you have that.

The pipeline is genuinely fast now — median end-to-end around 320ms, p99 under 600ms. I’m reasonably happy with it. But I’m also aware that “happy with it” is probably just “haven’t found the next bug yet.”