Your Kafka Infrastructure Just Became Optional

Five hops between an event and a queryable row. Snowflake says you only need one.

You’ve built the pipeline. Kafka cluster. Zookeeper (or KRaft if you’re current). Schema Registry. Kafka Connect with a Snowflake sink connector. An S3 landing zone. An ETL pipeline to pick up files and load them. Five separate systems, each with its own monitoring, scaling, failover, and billing, are required just to get an event from your application into a queryable table.

At Snowflake Summit 2026, two announcements landed that make this architecture optional: Snowflake Openflow (now GA across AWS, Azure, and GCP) and Datastream, a Kafka-compatible streaming endpoint native to Snowflake that eliminates the middleware entirely.

Here’s what changes, why it matters, and when you should (and shouldn’t) rip out your Kafka cluster.

The Problem: Real-Time Events Delayed by Infrastructure

The slide Snowflake showed at Summit tells the whole story:

EVENT (App/IoT/Web)
  → 1. Kafka Cluster
  → 2. Zookeeper & Schema Registry
  → 3. Kafka Connect
  → 4. S3/GCS Landing Zone
  → 5. ETL Pipeline
  → SNOWFLAKE

Each hop adds latency, cost, and operational burden. Each is a failure domain. Each requires a team (or at least a team member) to keep running.

The real cost isn’t compute; it’s attention. Every incident page at 2 am for a Kafka consumer lag spike, every Schema Registry compatibility check that blocks a deploy, every Connect connector that silently falls behind — that’s engineering time spent on plumbing instead of product.

The operational burden of each hop: Kafka Cluster (broker provisioning, partition rebalancing, disk, upgrades), Zookeeper/KRaft (quorum health, leader election, metadata drift), Schema Registry (compatibility, versioning, breaking-change coordination), Kafka Connect (configs, dead letter queues, offsets, plugins), Landing Zone (bucket policies, lifecycle rules, file accumulation), and ETL Pipeline (orchestration, retries, dedup, freshness checks).

Six systems. Six sets of credentials. Six monitoring dashboards. Six teams that can say “not my problem” when data stops flowing.

Infrastructure Complexity

The Solution: Two Products, One Goal

Snowflake attacked this problem from two angles simultaneously:

Openflow: The Integration Platform

Openflow is Snowflake’s fully managed data integration service, built on Apache NiFi. It’s been in preview for a while and went GA across all three major clouds in 2025.

What it gives you:

• Pre-built connectors for Kafka, MongoDB, BigQuery, Shopify, Veeva, Oracle, Salesforce, and dozens more
• Managed infrastructure — runs in Snowpark Container Services (SPCS) or your own VPC (BYOC)
• Snowpipe Streaming High Performance as the ingestion backend — 10 GB/s throughput, as low as 5 second end-to-end latency
• Single Message Transforms — filter, enrich, and reshape data before it lands
• Supports structured and unstructured data — JSON, Avro, Protobuf, images, audio, video, sensor data

For teams that still have Kafka (or MongoDB, or BigQuery) as a source, Openflow replaces the Connect + Landing Zone + ETL portion of the pipeline. Your Kafka cluster stays, but the five downstream hops collapse to one.

Datastream: The Kafka-Compatible Native Endpoint

Datastream is the more radical play. It’s a Kafka-compatible streaming endpoint built directly into Snowflake. Your producers speak the Kafka wire protocol — but there’s no Kafka cluster on the receiving end.

EVENT (App/IoT/Web)
  → DATASTREAM
  → Iceberg Tables / Snowflake Tables

Three components. One bill. Zero middleware.

Your application produces events using any Kafka client library. Datastream accepts them natively, handles partitioning and ordering, and writes directly to Iceberg tables or Snowflake tables. No brokers to provision. No partitions to rebalance. No consumer groups to manage.

Datastream Architecture

How Openflow Connectors Work

Openflow isn’t just a Kafka tool the Summit slide highlighted its broader connector ecosystem:

OpenFlow connectors, all generally available: Kafka (Apache Kafka, Confluent, AWS MSK), BigQuery (Google BigQuery), MongoDB (Atlas / self-hosted), Shopify (Shopify APIs), Veeva (Veeva Vault), Oracle (Oracle DB CDC), and Snowflake → Kafka (reverse ETL to Kafka topics).

Each connector uses the same deployment model:

1. Create an Openflow deployment (SPCS or BYOC)
2. Create a runtime within that deployment
3. Deploy the connector to the runtime
4. Configure source credentials and target table
5. Data flows

-- The Snowflake side: create the target table
CREATE OR REPLACE TABLE raw.events.kafka_events (
    record_metadata VARIANT,
    record_content VARIANT,
    _ingestion_time TIMESTAMP_NTZ DEFAULT CURRENT_TIMESTAMP()
);

-- Or use an Iceberg table for open format
CREATE OR REPLACE ICEBERG TABLE raw.events.kafka_events_iceberg (
    event_id STRING,
    event_type STRING,
    payload VARIANT,
    event_time TIMESTAMP_NTZ
)
CATALOG = 'SNOWFLAKE'
EXTERNAL_VOLUME = 'my_ext_volume'
BASE_LOCATION = 'events/';

The connector handles offset tracking, exactly-once delivery, and dead letter queue routing. You configure it through the Openflow canvas (a visual NiFi-based interface) or via APIs.

Authentication options: SASL (PLAIN, SCRAM-SHA-256, SCRAM-SHA-512, OAUTHBEARER), AWS MSK IAM, and mTLS. Covers every major Kafka deployment pattern.

Data formats: JSON (default), Avro, and Protobuf.

Snowpipe Streaming: The Engine Under the Hood

Both OpenFlow connectors and Datastream use Snowpipe Streaming High Performance as their ingestion backend. Understanding this layer matters because it defines the performance envelope:

Snowpipe Streaming High Performance: throughput up to 10 GB/s per table; latency as low as 5 seconds end-to-end; exactly-once delivery via offset token tracking; ordered within each channel; targets Snowflake tables and Snowflake-managed Iceberg tables (v2 and v3); automatic schema evolution; serverless pricing by uncompressed GB ingested.

The SDKs (Java, Python, Node.js) and REST API give you direct programmatic access to Snowpipe Streaming if you want to bypass both Kafka and Openflow entirely. For IoT edge devices or lightweight microservices, the REST API is the lightest path:

# Direct row-level ingestion via REST API
# No Kafka, no Openflow, no files — just HTTP
curl -X POST "https://<account>.snowflakecomputing.com/v1/streaming/channels/<channel>/rows" \
  -H "Authorization: Bearer $TOKEN" \
  -H "Content-Type: application/json" \
  -d '[
    {"event_id": "e-001", "event_type": "page_view", "ts": "2026-06-10T14:30:00Z"},
    {"event_id": "e-002", "event_type": "add_to_cart", "ts": "2026-06-10T14:30:01Z"}
  ]'

When to Use What

Not every team should rip out Kafka tomorrow. Here’s the decision matrix:

Match the tool to your situation: Kafka serving multiple consumers → keep Kafka, use the Openflow Kafka connector; Kafka only feeding the warehouse → replace with Datastream; brand-new pipeline → Datastream or the Snowpipe Streaming SDK; replicating from MongoDB/BigQuery/Oracle → Openflow connectors; under 1000 events/sec from IoT/edge → Snowpipe Streaming REST API; publishing from Snowflake to Kafka → Openflow Snowflake-to-Kafka connector.

Decision Tree

The Cost Equation

Let’s talk money. A typical mid-size Kafka deployment for warehouse ingestion:

Estimated monthly infrastructure cost: 3-broker Kafka cluster $1,200–$2,000; Zookeeper/KRaft $400–$600; Schema Registry $200–$400; Kafka Connect workers $600–$1,000; S3 storage + transfers $100–$500; ETL compute $500–$2,000 totaling roughly $3,000–$6,500/month, plus 10–20% of a senior engineer’s time (≈ $15K–$30K/month of attention).

Plus the hidden cost: 10–20% of a senior engineer’s time managing the above. At fully loaded cost, that’s $15K–$30K/month of engineering attention.

Snowpipe Streaming pricing is based on throughput credits per uncompressed GB ingested. For a team ingesting 100 GB/day, the Snowflake-side cost is a fraction of the cost of the infrastructure it replaces. And it’s serverless: no clusters to right-size, no capacity planning.

The catch: if Kafka serves purposes beyond Snowflake ingestion (event sourcing, CQRS, cross-service communication), you can’t eliminate it. You can only simplify the Snowflake ingestion path.

Deployment Models: SPCS vs. BYOC

OpenFlow gives you two deployment options:

SPCS runs in Snowpark Container Services across AWS/Azure/GCP with native Snowflake auth and managed scaling, best for standard use cases and quick setup. BYOC runs in your own AWS VPC with Secrets Manager/Vault, PrivateLink and custom peering, and self-managed scaling, best for private sources, compliance, and data residency.

For most teams, SPCS is the right starting point. BYOC makes sense when your data sources are in a private network that can’t be reached from Snowflake’s infrastructure, or when compliance requires data to never leave your VPC.

What This Means for the Streaming Ecosystem

Snowflake isn’t killing Kafka. Kafka remains the dominant event streaming platform for cross-service communication, event sourcing, and complex event processing.

What Snowflake is killing: the “Kafka as a conduit to the warehouse” pattern. The use case where Kafka exists primarily (or solely) to buffer events before loading them into an analytical store — that pattern is now redundant.

The industry is converging on a clear split:

Where each tool wins: service-to-service event communication → Kafka (or Pulsar, or Kinesis); event sourcing / CQRS → Kafka; complex event processing with stateful operators → Kafka Streams / Flink; getting events into your analytical warehouse → Snowpipe Streaming / Datastream / OpenFlow; CDC from databases into Snowflake → OpenFlow connectors.

The Kafka ecosystem won’t shrink. But Kafka clusters that exist only as expensive, over-engineered S3 writers — those are going away.

Ecosystem Diagram

Limitations (Because You Should Know Before You Commit)

• Datastream is a new announcement. Check Snowflake docs for current availability status and regional support before building production pipelines on it.
• OpenFlow autoscaling is not supported for the Kafka connector. You set a fixed node count. Under-provision and you get backpressure.
• OpenFlow BYOC is AWS-only. If your sources are in Azure or GCP private networks, you need the SPCS deployment with external access integrations.
• Schema evolution for Iceberg tables is not supported in the OpenFlow Kafka connector. If your schemas change frequently, target Snowflake-native tables instead.
• Only one SPCS deployment per account. You separate workloads by creating multiple runtimes within that deployment, not multiple deployments.
• Snowpipe Streaming latency is “as low as 5 seconds”; it is not guaranteed to be 5 seconds. Actual latency depends on ingestion volume, channel count, and table configuration.
• Kafka connector requires Kafka 0.10.0.0+. Legacy Kafka versions are not supported.

Quick Reference

The Bigger Picture

Data infrastructure is consolidating. The era of “best-of-breed tool for every hop” created resilient but expensive architectures. The next era rewards platforms that can collapse multiple hops into one — without sacrificing reliability.

Snowflake’s bet: if the warehouse can speak Kafka protocol natively, the warehouse becomes the streaming endpoint. No translation layer needed. No impedance mismatch. No six-system Rube Goldberg machine between an event and a query.

The teams that will benefit most aren’t the ones with complex multi-consumer Kafka deployments. They’re the ones who built Kafka because it was the only reliable way to get real-time data into their warehouse — and now have a simpler option.

Closing Image

Audit your Kafka deployment this week. If more than 50% of your topics exist solely to feed your warehouse, OpenFlow, or Datastream, just made half your infrastructure redundant.