GCP-PDE — Building the Cymbal System That Can Change Without Breaking — Flexibility & Portability (Section 1.3)

A system that grows with the business, not against it.

I once watched a data team spend three months trying to migrate a pipeline to a new cloud provider. Not because the technology was hard. Because nobody had designed for the possibility that they might ever need to move. Every layer was hardcoded — proprietary formats, vendor-specific APIs, region-locked storage — and each one was a new toll booth on the way out.

That’s what a system without portability looks like. It works perfectly until the business changes direction, and then it becomes a cage.

Section 1.3 is the exam’s way of asking: can you build a Cymbal data system that adapts to what the business needs tomorrow — not just what it needs today? Flexibility isn’t about predicting the future. It’s about not closing doors you’ll wish you’d left open.

To make this concrete, we’ll follow Cymbal’s Global Expansion Programme — the company is growing from a US-only retailer into a multi-region, multi-cloud organisation with stores in Germany, Singapore, and Brazil. Each market comes with its own regulatory requirements, latency constraints, and existing vendor relationships. The architecture we build today has to serve all of them.

01 — Reading the Map Before You Build the Road: Mapping Business Requirements to Architecture

Think of business requirements like a city’s transit map. Before you lay a single rail, you need to know where people are going today — and where the city planners say they’ll want to go in five years. Building only for today means tearing up the track tomorrow. Building only for tomorrow means a system nobody can use yet. The exam tests whether you can hold both at once.

Current vs future state mapping

• Start by documenting what Cymbal actually needs right now: batch inventory syncs, a US-only analytics warehouse, a single Pub/Sub topic for store events.
• Then document the future state: real-time personalisation across three continents, data residency compliance in the EU and Brazil, an ML-powered demand forecasting model.
• The gap between these two maps is your architecture brief — every tool choice should close that gap without over-engineering today.

Designing for schema evolution

• Cymbal’s product catalogue will grow. New attributes will be added, old ones deprecated. A rigid schema that requires a full pipeline rewrite every time a field changes is a liability, not an asset.
• BigQuery’s native support for schema auto-detection and flexible column additions means you can add a product_sustainability_score field next quarter without touching the pipeline.
• Avro and Protobuf, used as message formats in Pub/Sub, support schema evolution natively — new fields are ignored by old consumers, old fields have defaults for new producers. This is the right serialisation choice for a system that will change.

Designing for volume growth

• Cymbal processes 10,000 store events per day today. After expanding to Singapore and Brazil, that becomes 500,000. An architecture that requires manual scaling interventions at every order of magnitude is fragile.
• Dataflow’s autoscaling automatically adjusts worker count to match incoming volume — no operator needs to set a dial at 2 AM when Singapore opens for business.
• BigQuery’s serverless model means Cymbal’s analysts query the same table whether it holds 10 GB or 100 TB — no cluster to resize, no performance tuning required.

Designing for new use cases

• Today Cymbal uses BigQuery for historical sales analytics. Next year, they want to add BigQuery ML for demand forecasting. The year after, a Vertex AI pipeline for product recommendations.
• If the data is already clean, catalogued, and accessible in BigQuery, each new use case is additive — not a rebuild from scratch.
• The principle here: design the data layer to be use-case agnostic. The consumers change; the foundation doesn’t.

💡 Exam Tip: When a scenario describes a company “planning to expand” or “anticipating new workloads,” the exam is testing whether you choose a rigid point solution or a composable, scalable architecture.

Schema evolution → Avro/Protobuf in Pub/Sub + BigQuery schema flexibility

Volume growth without manual intervention → Dataflow autoscaling + BigQuery serverless

Future use cases on existing data → clean, catalogued data in BigQuery accessible to BigQuery ML and Vertex AI

02 — The Passport System: Data and Application Portability

Cymbal used to live entirely inside GCP. Now it has a German subsidiary already committed to Azure for its ERP system, and a Brazilian market where a local regulation requires a local cloud presence. Portability isn’t a nice-to-have anymore — it’s a business requirement. The exam will test whether you can architect a system that can move, share, and exist across boundaries.

Multi-cloud data sharing with BigQuery Omni

• BigQuery Omni lets Cymbal run BigQuery SQL queries directly against data stored in AWS S3 or Azure Blob Storage — without moving the data into GCP first.
• For the German subsidiary’s sales data sitting in Azure, Cymbal’s central analytics team can query it from BigQuery as if it were local — no pipeline, no copy, no egress cost.
• This is the exam’s answer to “how do you analyse data across clouds without data duplication.” The data stays where it lives; the compute comes to it.

Open formats as your portability insurance

• The single most powerful portability decision Cymbal can make is choosing open storage formats: Parquet and ORC for columnar analytics data, Avro for streaming and serialisation, JSON for semi-structured event data.
• Data stored in Parquet on Cloud Storage can be queried by BigQuery today, by Spark on Dataproc tomorrow, by a Databricks cluster on Azure next year — without conversion.
• Proprietary formats are the lock-in. Open formats are the escape hatch. The exam knows this distinction.

Apache Beam and Dataflow as portable compute

• Apache Beam is the open-source SDK that Dataflow runs on. The same pipeline code that runs on Dataflow (GCP) can run on Apache Flink or Apache Spark on any cloud with minimal changes.
• If Cymbal ever needs to move a Dataflow pipeline to run on-prem or on another cloud, the Beam abstraction layer means the processing logic is already portable — only the runner changes.
• AWS folks: think of this as writing your ETL in Spark so it runs on EMR today and Azure HDInsight tomorrow. Same principle, different ecosystem.

Data residency in a multi-cloud world

• Germany requires customer data to stay within German borders. Brazil’s LGPD (Brazil’s GDPR equivalent) requires Brazilian customer data to stay within Brazil.
• For GCP-native data: store in europe-west3 (Frankfurt) for Germany and southamerica-east1 (São Paulo) for Brazil — single-region datasets, not multi-region.
• For data shared with the Azure ERP in Germany: BigQuery Omni with the Azure West Europe region keeps data within EU jurisdiction while enabling cross-cloud analytics.
• The principle: residency is about where data is stored at rest, not where queries are issued from.

Application portability with containers and Kubernetes

• Cymbal’s data processing applications — custom Dataflow templates, API services, ML model serving — should be containerised with Docker and deployed via Google Kubernetes Engine (GKE).
• A containerised application can move between GKE (GCP), EKS (AWS), and AKS (Azure) with configuration changes, not code rewrites.
• Artifact Registry stores Cymbal’s container images centrally — the same image deployed in Singapore today can be deployed in Brazil tomorrow without rebuilding.

💡 Exam Tip:

“Query data in AWS S3 or Azure from BigQuery without moving it” → BigQuery Omni

“Store data in a format that works across cloud tools” → Parquet / ORC / Avro on Cloud Storage

“Make pipeline code portable across cloud runners” → Apache Beam (Dataflow SDK)

“Data must stay in [specific country] — multi-cloud scenario” → single-region dataset in the named region + BigQuery Omni for cross-cloud queries

“Move application workloads between clouds” → containerisation with Docker + GKE

03 — The Library System: Data Staging, Cataloging, Profiling, and Discovery

Imagine Cymbal’s data estate as a library that has been collecting books for ten years — but nobody ever built a catalogue. Analysts wander the shelves guessing. Engineers copy tables they’re not sure already exist. ML teams build features on columns nobody can explain. The library is full of knowledge but functionally useless.

Data governance is the act of building the catalogue, hiring the librarian, and writing the rules about who can check out which books. The exam tests whether you know which GCP tool plays which role in this library.

Data staging — the loading dock

• Before data enters Cymbal’s governed data estate, it lands in a staging area first — typically raw Cloud Storage buckets or staging BigQuery datasets.
• Staging gives you a checkpoint: validate, inspect, and apply transformations before data reaches the curated layer that analysts trust.
• Think of it as the library’s receiving dock — books arrive, get catalogued and labelled, and only then go on the shelves.

Dataplex — the librarian and the governance system

• Dataplex is GCP’s unified data governance platform. It organises Cymbal’s data across Cloud Storage, BigQuery, and other sources into logical Lakes and Zones — without physically moving anything.
• A Lake is the top-level domain (e.g., “Retail Operations”). Inside it, Zones separate data by maturity: a Raw Zone for unprocessed ingestion data, a Curated Zone for cleaned and validated data, a Consumption Zone for analytics-ready datasets.
• Dataplex automatically discovers new assets, applies metadata tags, enforces data quality rules, and tracks lineage — all from one control plane.

Data Catalog — the card index

• Data Catalog is GCP’s metadata management service. Every BigQuery table, Cloud Storage file, and Pub/Sub topic Cymbal owns can be registered here with descriptions, tags, ownership information, and schema details.
• An analyst searching “customer transaction” finds exactly which tables exist, who owns them, what the columns mean, and when they were last updated — without filing a ticket to the data engineering team.
• Data Catalog integrates with Dataplex — when Dataplex discovers a new asset, it automatically populates Data Catalog. You don’t maintain two systems separately.

Data profiling — the quality scanner

• Profiling means systematically measuring the characteristics of a dataset: how many nulls in each column, what’s the value distribution, are there statistical outliers, what are the min/max/mean values.
• Dataplex’s built-in data profiling runs automatically on a schedule and produces a quality report for every managed table — no custom scripts required.
• For Cymbal’s product catalogue, profiling might reveal that 40% of the product_weight_kg column is null — something no schema validation would catch, but something that would silently break a downstream shipping cost calculation.

Data discovery — finding what you don’t know you have

• As Cymbal grows, new tables and datasets are created constantly. Without discovery, the same data gets re-ingested by three different teams who didn’t know it already existed.
• Dataplex’s auto-discovery scans Cloud Storage and BigQuery on a schedule, identifies new assets, infers their schema, and registers them in Data Catalog automatically.
• The AWS equivalent here is AWS Glue Data Catalog with Glue Crawlers — same pattern of automated discovery feeding a centralised metadata store.

Data lineage — following the bread crumb trail

• Lineage tracks where data came from, what transformations it went through, and where it ends up. When Cymbal’s finance team asks “where does this revenue number come from?”, lineage gives you the full answer in a visual graph.
• Dataplex captures lineage automatically for BigQuery operations and Dataflow pipelines — no manual annotation needed.
• When a source table changes its schema, lineage immediately shows you every downstream table and report that depends on it — so you know what breaks before it breaks.

Policy Tags for governed discovery

• Not every column in Cymbal’s catalogue should be discoverable by everyone. Salary data, customer PII, and payment card fields are tagged with Policy Tags that restrict both access and visibility.
• A data scientist searching Data Catalog sees that the customer_email column exists — but can't query it without the Fine-Grained Reader role on that specific tag.
• This is governance that works at the column level, not just the table level — discovery without over-exposure.

💡 Exam Tip: These four concepts — staging, cataloging, profiling, discovery — map to distinct GCP tools. Know which is which:

Staging → Cloud Storage raw bucket or staging BigQuery dataset

Cataloging + Discovery → Data Catalog (metadata) + Dataplex (governance + auto-discovery)

Profiling → Dataplex data profiling (automated quality statistics)

Lineage → Dataplex lineage (auto-captured for BigQuery + Dataflow)

Column-level governed access → Policy Tags in Data Catalog

Practice Questions

Read the scenario carefully before jumping to the answer. The wrong choices are usually well-reasoned — just for the wrong problem.

Q1 — Schema Evolution

Cymbal’s product team wants to add a new sustainability_rating field to the inventory events flowing through Pub/Sub into BigQuery. The pipeline must continue processing old events (without the new field) and new events (with the field) simultaneously without a full pipeline rewrite. Which design choice best supports this?

• A. Use JSON serialisation for Pub/Sub messages and add a NOT NULL constraint on the new field in BigQuery
• B. Use Avro serialisation for Pub/Sub messages with a default value on the new field; use BigQuery’s schema auto-detection for new columns
• C. Create a separate Pub/Sub topic for new events and a second Dataflow pipeline
• D. Use Protocol Buffers with a required field for sustainability_rating

Answer: B

• Avro supports schema evolution natively — new fields with default values are backward-compatible, meaning old consumers ignore the field and new producers include it.
• BigQuery’s schema auto-detection (or explicit schema update) adds the new column to the existing table without affecting historical rows.
• JSON (A) is flexible but NOT NULL on a new field would immediately reject all old events that lack it — wrong direction.
• A separate topic and pipeline © works technically but creates operational duplication for a problem schema evolution was designed to solve.
• Protobuf with a required field (D) is the worst choice — required fields break backward compatibility by definition; any old message without the field is immediately invalid.

Q2 — Multi-Cloud Analytics

Cymbal’s German subsidiary stores its ERP sales data in Azure Blob Storage. Cymbal’s central analytics team in the US needs to run BigQuery SQL analytics across both GCP-native inventory data and this Azure sales data — without duplicating data or building a migration pipeline. What is the correct solution?

• A. Use Cloud Data Fusion to replicate Azure data into BigQuery on a nightly schedule
• B. Use BigQuery Omni to query the Azure data in place from a BigQuery analytics hub
• C. Export Azure data to Cloud Storage and load it into BigQuery using a transfer service
• D. Use Dataflow to stream Azure Blob Storage events into BigQuery in real time

Answer: B

• BigQuery Omni is built exactly for this scenario — run BigQuery SQL against data in Azure or AWS without moving it into GCP.
• Data Fusion (A) and scheduled exports © both involve data movement and duplication — extra cost, extra latency, extra compliance risk.
• Dataflow streaming (D) is the right tool for real-time event processing, not for querying existing historical data sitting in a blob store.

Q3 — Data Residency in Multi-Cloud

Cymbal is expanding into Brazil. Brazilian law (LGPD) requires that all Brazilian customer data be stored within Brazil’s geographic borders. Cymbal’s current setup uses a US multi-region BigQuery dataset for all markets. What is the minimum correct change?

• A. Enable VPC Service Controls on the US dataset to prevent data from leaving GCP
• B. Create a separate BigQuery dataset in southamerica-east1 for Brazilian customer data
• C. Encrypt the Brazilian customer data with a CMEK key stored in a Brazilian KMS location
• D. Set the BigQuery dataset location to southamerica-east1 and use Dataplex to sync with the US dataset

Answer: B

• LGPD requires data to physically reside within Brazil — southamerica-east1 (São Paulo) is the correct single-region choice.
• VPC Service Controls (A) prevents data exfiltration — it doesn’t control where data is physically stored at rest.
• CMEK in a local KMS location © controls who holds the encryption key, not where the data lives. A Brazilian KMS key does not mean the data is stored in Brazil.
• Dataplex sync (D) would replicate the data, which means it exists in both locations simultaneously — potentially a compliance violation for LGPD.

Q4 — Cataloging vs Profiling vs Discovery

After Cymbal’s data engineering team creates dozens of new BigQuery datasets during the global expansion, business analysts are struggling to find the right tables and understand what the columns mean. Data engineers are getting tickets asking “what does inv_qty_adj mean?" several times a week. Which combination of GCP capabilities best solves this?

• A. Dataform data quality tests + Cloud Monitoring dashboards
• B. Data Catalog business metadata + Dataplex auto-discovery
• C. BigQuery column descriptions + Cloud Logging
• D. Dataplex data profiling + VPC Service Controls

Answer: B

• Data Catalog lets data engineers add plain-English descriptions, ownership tags, and business context to every table and column — so inv_qty_adj becomes "Inventory Quantity Adjusted — net units after returns and damage write-offs."
• Dataplex auto-discovery ensures that new datasets are automatically registered in Data Catalog without manual intervention — the catalogue stays current as the estate grows.
• BigQuery column descriptions © live inside BigQuery only — they don’t surface in a searchable catalogue accessible to all analysts.
• Data profiling (D) measures data quality statistics — it tells you that 15% of inv_qty_adj is null, not what the column means.

Q5 — Pipeline Portability

Cymbal’s data engineering team builds a Dataflow pipeline to process store inventory events. Six months later, Cymbal acquires a company that runs all its infrastructure on AWS and needs the same pipeline logic to run there temporarily during migration. Which design decision made at the start would make this migration easiest?

• A. Writing the Dataflow pipeline using only GCP-native Dataflow transforms and I/O connectors
• B. Building the pipeline using the Apache Beam SDK with abstracted I/O connectors, deployable on any Beam runner
• C. Using Cloud Data Fusion to build the pipeline visually so it can be exported
• D. Using BigQuery scheduled queries instead of Dataflow for the transformation logic

Answer: B

• Apache Beam is the portability layer — the same pipeline code runs on Dataflow (GCP), Apache Flink, Apache Spark, or any Beam-compatible runner. Only the runner configuration changes during migration.
• GCP-native transforms (A) are the opposite of portable — they couple the pipeline to Dataflow’s proprietary extensions.
• Cloud Data Fusion © is a visual ETL tool for GCP — its pipelines are not natively exportable to AWS runners.
• BigQuery scheduled queries (D) are even more tightly coupled to GCP — they run nowhere else.

Q6 — Data Lineage

Cymbal’s finance team notices that a key revenue metric in their weekly dashboard has changed unexpectedly. They suspect a source table schema change broke a transformation somewhere in the pipeline, but with hundreds of tables and pipelines, they can’t tell where. Which GCP capability helps them trace the root cause?

• A. Cloud Monitoring alerts on BigQuery query errors
• B. Dataplex data lineage graph for the revenue metric’s source tables
• C. Dataform test failures in the transformation DAG
• D. Data Catalog search for tables tagged “revenue”

Answer: B

• Dataplex lineage shows exactly where data came from, which transformations it passed through, and which downstream tables and dashboards depend on it — a visual map from source to output.
• Cloud Monitoring (A) tells you that errors occurred, not which upstream change caused them.
• Dataform tests © catch failures during transformation runs — but if the schema change didn’t trigger a test failure, it wouldn’t surface here.
• Data Catalog search (D) helps you find tables — it doesn’t show dependencies between them.

Q7 — Staging and the Data Maturity Model

Cymbal’s data team wants to build a governed BigQuery environment where raw ingested data is never directly accessible to analysts, cleaned data is accessible to internal teams, and fully anonymised aggregated data is accessible to external partners. Which architecture best represents this?

• A. Three separate GCP Projects with IAM-controlled access between them
• B. A single BigQuery dataset with row-level security policies for each user group
• C. A Dataplex Lake with three Zones — Raw, Curated, and Consumption — each with separate IAM policies
• D. Three Cloud Storage buckets with Dataflow pipelines moving data between them

Answer: C

• Dataplex Lakes and Zones are designed exactly for this maturity model: Raw Zone (data engineers only), Curated Zone (internal analysts), Consumption Zone (external partners) — with IAM policies enforced at each zone boundary.
• Separate projects (A) create hard organisational boundaries but introduce cross-project complexity and billing overhead that Dataplex’s zone model avoids.
• A single dataset with row-level security (B) doesn’t separate the raw and curated layers — everything is in one place, which means a misconfigured policy could expose raw data.
• Cloud Storage buckets with Dataflow (D) is an anti-pattern here — it stores analytical data outside BigQuery, losing all the governance, cataloguing, and SQL querying benefits.

A Final Reflection

There’s a certain arrogance in designing systems that assume the future will look exactly like the present. Cymbal wasn’t always a global retailer. It started as a US-only store, and every technical decision it made without portability in mind became a wall it had to tear down later.

What flexibility and portability really ask of you is humility. Humility to say: I don’t know which cloud this will run on in three years. I don’t know what new regulations will emerge. I don’t know which use cases the business will think of next. And then to design a system that can absorb those unknowns without collapsing.

Open formats, abstract compute layers, centralised catalogues, governed zones — these aren’t overengineering. They’re the difference between a system that serves the business and a system the business eventually works around.

That’s what we’re building.

Section 1.4 covers Designing data migrations— coming next in the series.