Google Cloud Next ’26, Data Infrastructure Perspectives

“AI” was the headline at Google Cloud Next ’26, but underneath the AI announcements was a sustained data infrastructure story: Google is elevating Apache Iceberg to a first-class open table format for cross-engine and cross-cloud interoperability — alongside, not in place of, BigQuery’s native storage — making Knowledge Catalog the metadata standard, and positioning BigQuery + Managed Spark as complementary engines on top. The cumulative direction strengthens the case for moving datasets off warehouse-native storage and onto open formats, makes a metadata catalog a useful upstream feed for any data platform, and opens a credible path from data engineering through ML on a single substrate.

The Frame: Why Data Matters More Than Ever

AI was the dominant theme — new TPUs, plain-English analytics, AI-native developer experience — but Google kept returning to a basic point: AI is only as good as its data. Garbage in, garbage out. From there, the data infrastructure story splits into two fronts. Data quantity is about reach: GCP already holds vast amounts of customer data, but more lives outside it (most notably in AWS), and Google is pushing for cross-cloud reach so customers can use GCP’s AI without first moving data. Data quality is about form: data has to be in a shape engines and agents can interpret, with rich metadata for context, which Google is addressing through standardization (Iceberg), a centralized metadata layer (Knowledge Catalog), and automatic metadata extraction at the storage tier (GCS auto-annotation). Most of the data-platform announcements line up cleanly under one of these two fronts.

Data Quantity: Reach Across Clouds

GCP wants to be a consumer of customers’ existing data wherever it lives, not a destination customers have to migrate to first. Three announcements drive this. The Cross-Cloud Lakehouse on Iceberg lets data stay in S3 today (Azure later in 2026) while BigQuery queries it via the Iceberg REST catalog — no migration, no egress, no copy. Cross-Cloud Caching materializes a cache near BigQuery on the first read so repeat egress is avoided and follow-on queries accelerate, directly attacking the cost objection most often raised against cross-cloud queries. Lakehouse Catalog Federation adds unified discovery and zero-copy access across AWS Glue, Databricks, Snowflake, SAP, Confluent Tableflow, and many others.

Data Quality: Standardization and Context

Iceberg as the open table format standard. The Iceberg table format is the de-facto industry standard when interoperability between compute engines matters, and Google has invested heavily in it since 2025. The headline change at Next ’26 is REST catalog Iceberg tables that can be read and written by BigQuery, Spark, Trino, Flink, and many other third-party engines (Databricks and Snowflake in Preview), backed by a managed Iceberg storage and REST catalog with automatic table maintenance, multi-table transactions, and history-based optimization.

Knowledge Catalog as the metadata standard. Knowledge Catalog, formerly Dataplex, is a centralized metadata layer that aggregates metadata across BigQuery tables, Iceberg tables, GCS objects, and external sources, giving users and agents one place to discover and reason about the data estate. It collects metadata, lineage, and data-quality signals across Google Cloud services (BigQuery, Cloud Storage, Vertex AI, Spanner, Looker, and others), third-party Iceberg catalogs via the Iceberg REST Catalog (AWS Glue, Databricks Unity, Snowflake Horizon, all in Preview), and enterprise systems such as Salesforce Data360, SAP, ServiceNow, and Workday via context federation. It also ingests metadata produced upstream — for example, the auto-annotations GCS now generates on newly landed objects — so storage-tier extraction flows in without manual curation. The whole layer is designed to feed agent systems with always-on, access-controlled context.

Compute on the Lakehouse

Once data is in Iceberg, the compute story stops being “Spark vs. BigQuery” and becomes “use whichever fits the workload, on the same data.” On the Spark side, Lightning Engine for Apache Spark is a C++ native execution engine with vectorized execution, intelligent caching, optimized GCS I/O, and a faster Parquet reader, with zero-copy memory management between C++ and the JVM, claiming up to 4.9x over OSS Spark and 2x price-performance over leading alternatives, with no code changes for Iceberg / Parquet workloads. On the BigQuery side, BigQuery on Iceberg is getting the same kind of efficiency work (better GCS I/O, better Parquet reader) along with streaming into Iceberg tables, CDC between Iceberg and BigQuery, and a first-class Iceberg UX in BigQuery Studio. BigQuery’s broader engine improvements include Advanced Runtime delivering ~35% year-over-year performance and Fluid Scaling adding per-second billing for spiky agentic workloads. The combined effect: the historical penalty for putting data in Iceberg-on-GCS instead of BigQuery-native storage is shrinking, and the gap is expected to keep closing.

Storage Substrate: GCS Upgrades

GCS itself was tuned for AI workloads. Objects can be auto-annotated via Smart Storage and the Object Context API — pretrained models generate image labels, object detection with confidence scores, and objectionable-content flags as soon as files land — and Rapid storage and Rapid cache push I/O performance forward for both training and serving paths. Because Iceberg tables live on GCS, the same I/O gains compound with the BigQuery-on-Iceberg and Lightning Engine optimizations to lift Iceberg query performance too.

From Data to ML on One Substrate

The compute tier underneath Iceberg has quietly become a viable ML platform too. Managed Service for Apache Spark (the rebrand of Dataproc) supports GPUs — A100 and L4 — with the Spark runtime shipping Spark RAPIDS and XGBoost on every node. A workload optimized for data processing can flip on GPU without leaving the product or recasting the cluster shape. The result is that the same Iceberg + Spark substrate that handles ingestion and transformation can extend, with GPU enabled, into feature engineering, training, and batch inference — on the same data, in the same product.

Unified Developer Experience

GCP is collapsing the analyst, data engineer, and ML practitioner experience into one Python environment. Plain-English analytics on top of Knowledge Catalog runs inside BigQuery Studio in place of hand-written SQL. The notebook surface — Colab Enterprise or VSCode powered by the Google Cloud Data Agent Kit — covers ingestion, transformation, and analysis with BigQuery and Managed Spark usable interchangeably from the same Python session, and the same environment is meant to carry through into feature engineering, backfills, model training, and evaluation.

What This Could Mean

Several threads run in this same direction, and Next ’26 largely confirms it.

Open formats loosen vendor and cloud lock-in. Underneath the specific launches is a more durable shift, and it has little to do with any one feature. For years the price of a managed data platform was lock-in: data sat in a proprietary format, on a single vendor’s cloud, readable only by that vendor’s engines, and leaving meant a full copy-out and rebuild that few teams ever attempt. An open table format changes that. Once the storage format is open and any compliant engine can read and write it, the data stops being captive — engines become interchangeable, chosen per workload rather than committed to for the life of a table, and the format is no longer pinned to one provider’s infrastructure. Compatibility and cloud independence are two views of the same gain: the data stays portable, and both the engine and the cloud become decisions that can be revisited. The headline AI capabilities will keep turning over, but the data beneath them need not. Moving that foundation out of a proprietary store and into an open format is the slower decision — and the one that compounds for years.

GCP is becoming Iceberg + Spark first, fast. Spark and BigQuery can be treated as complementary rather than competing. I can pick the engine that fits the workload — Spark for heavy transformations and Python-centric pipelines, BigQuery for SQL-driven analytics — without paying a storage-layer tax to switch between them. The unified Python notebook environment goes one step further: in a single session, users can hit BigQuery and Spark interchangeably against the same Iceberg data, so the engine becomes a choice made cell by cell, not an architectural commitment that spans the whole pipeline. Given the roadmap of efficiency improvements Google has signaled on top of Iceberg, computation over Iceberg should keep getting cheaper and faster, which weakens one of the historical arguments for keeping data in engine-native formats.

Iceberg + Knowledge Catalog will significantly boost AI agents. AI use cases tend to span two flavors. The first is AI built into a product, where models are trained and served against curated datasets along well-defined pipelines. The second is agentic AI used across a business: ad-hoc data analysis, research, and other workflows where people hand questions off to agents that have to reason over the data estate themselves. The Iceberg + Knowledge Catalog story matters for both, but it is especially load-bearing for the second flavor, because those agents need broad, standardized context across many datasets at query time rather than a hand-curated training set.

Already running my own data catalog, I need not switch wholesale. The realistic path is to treat a standard like Knowledge Catalog as a source to pull from — auto-extracted metadata, lineage, quality signals, standardized definitions — to enrich the catalog I already have, while keeping the business-specific features that only make sense in my own system. Over time that gives my agents both the standardization the vendor produces and the context only I can produce.

Appendix: References

• Next ’26 Storage Announcements
• What’s New in the Agentic Data Cloud
• Introducing the Google Cloud Knowledge Catalog
• The Future of Data Lakehouse for the Agentic Era
• Unveiling New BigQuery Capabilities for the Agentic Era
• Understand Catalog Types

originally from https://uncommonstories.substack.com/p/google-cloud-next-26-data-infrastructure