From Kafka to Iceberg with No Code — Introducing ItdaStream Streaming

Produce a message to a topic, and it lands in an Iceberg table — without writing a single line of code.

ItdaStream is a distributed streaming platform that speaks the Apache Kafka wire protocol. You keep your existing Kafka clients unchanged, but the platform disaggregates compute (brokers) from storage (S3) for unlimited retention at a fraction of the cost. And as of 1.0.0, a Flink-style streaming engine runs inside the brokers, landing topic data into an Apache Iceberg lakehouse with exactly-once semantics in real time.

This post first recaps ItdaStream’s fundamentals, then dives into the new streaming capability.

1. First, the fundamentals

Before the streaming story, here’s the foundation ItdaStream stands on.

• 100% Kafka protocol compatibility — implements the Kafka API (including transactions), so your existing clients and ecosystem work as-is. No code changes to migrate.
• Compute–storage disaggregation (stateless brokers + S3) — brokers are stateless. Adding or removing a node triggers no large-scale data reshuffling, and data is retained indefinitely on S3 with no local disk ceiling.
• Tiered storage — recent data in memory/SSD (Hot tier), older data on S3 (Cloud tier). Asynchronous flush and tail-read optimization keep real-time latency even on an S3 backend.
• Exactly-once transactions — read_committed isolation and a ZooKeeper-backed transaction state machine.
• Consumer group coordination — Join/Sync/Heartbeat with automatic rebalancing.
• Enterprise security — KMS envelope encryption (AES-GCM 256), AWS IAM-compatible access control (users/groups/JSON policies), SASL/PLAIN authentication and TLS.
• Operability — high-performance NIO networking, intelligent segment caching, log retention & compaction, real-time monitoring, network compression, backup/restore, and a built-in Admin UI.

On top of this, 1.0.0 targets one of the nastiest failure modes in distributed systems.

NIC silent-fail safety

Imagine a peer whose NIC dies after the socket was opened, but whose ZooKeeper session hasn’t expired yet. To ZooKeeper and to every neighbor, the node still looks alive. The OS won’t surface the half-open connection for tens of seconds — sometimes minutes — while it drains its retransmit budget, so RPCs routed to that peer hang silently. Membership-driven failover never fires because membership is healthy — and a single sick peer can make the whole cluster look wedged.

ItdaStream’s internal NIO plane closes this gap on both ends of every RPC. It is non-blocking for its entire lifetime with a per-call Selector, bounds connect/write with a 10-second deadline (dropping and re-minting the connection on timeout), and enables SO_KEEPALIVE/TCP_NODELAY so the OS itself eventually flags long-dead peers. A single sick peer can no longer permanently occupy a thread in the master's heartbeat loop or a worker's sync loop.

2. The main event: streaming (Kafka → Iceberg)

Now the streaming engine. The controller (leader) broker acts as the master, and the other brokers act as workers — streaming runs on the very cluster that stores your topics, so there’s no separate stream-processing cluster (e.g. Flink) to operate.

There are two ways to use it:

• No-code — in the Admin UI, just pick a topic and a target Iceberg table.
• SDK — submit a Java job with transforms (filter/select/map/flatMap) and any number of sinks.

2.1. No-code: produce to a topic, get Iceberg

The most common pipeline — “messages in a topic accumulate in an Iceberg table” — needs no code at all. The flow is this simple:

Producer  →  topic: events (JSON/Avro)  →  ItdaStream streaming engine (exactly-once)  →  Iceberg table

You configure it once in the Admin UI:

1. Register a connection — add an Iceberg connection (catalog URI, warehouse, S3 credentials) to the Connection Registry. Credentials are stored once; jobs reference them only by connectionId (no secrets in the job).
2. Pick topic & format — the source topic and json/avro.
3. Set parallelism — the number of consumer threads (= the job’s consumer-group size). Keep it ≤ the partition count so every thread stays busy.
4. Target table — namespace.table. For change streams, set upsert keys for idempotent updates.

From then on, every message produced to the topic flows into Iceberg automatically.

Submitting via REST, the job spec looks like this (the UI produces the same spec):

{
  "name": "events-to-iceberg",
  "parallelism": 4,
  "kafka": { "topic": "events", "format": "json" },
  "sink": {
    "type": "table",
    "connectionId": "prod-iceberg",
    "table": "analytics.events",
    "upsertKeys": ["id"]
  },
  "commitIntervalMs": 5000,
  "checkpointIntervalMs": 5000
}

Note: the sink auto-creates the table (inferring the schema from the first batch), but the Iceberg namespace must already exist.

2.2. Master/worker architecture

How does the engine actually run?

• Master (the controller). Watches /streaming/jobs, splits a job's parallelism into a per-broker thread count {brokerId: threadCount}, and writes it to /streaming/assignments/<jobId>. It also runs one CheckpointCoordinator per job, sending checkpoint barriers to the workers over internal NIO.
• Worker (every broker). Converges its running executor threads to its assigned count. Each thread is one consumer in the group stream-<jobId> that deserializes → transforms → writes to the sink.

So parallelism = consumer-group size. The topic partitions are distributed across the threads by ItdaStream’s existing group coordinator, which rebalances as brokers join and leave. The control plane (assignments) flows through ZooKeeper, the checkpoint barriers through internal NIO, and the data plane is the consumer group.

2.3. How exactly-once is guaranteed

For transactional sinks (Iceberg/JDBC), the poll loop never advances offsets first. At each checkpoint it achieves exactly-once in this order:

1. Commit the sink — commit the Iceberg snapshot (or JDBC transaction) first.
2. Snapshot offsets — write the Kafka offsets to the S3 ExchangeManager (KMS-encrypted).
3. Commit offsets — only now advance the Kafka offsets (the commit point).
4. Ack — report completion to the controller’s coordinator.

Because offsets advance only after the sink commit, a restart seeks the consumer to the last completed checkpoint's offsets in S3 and resumes exactly — no loss, no duplicates. (With upsert keys, even boundary reprocessing converges to the latest value.)

If a worker dies, its partitions rebalance to other workers; if the controller dies, a new one re-reads /streaming/jobs and resumes assignment + checkpoint coordination. The data plane keeps running regardless of who is master.

2.4. Not just Iceberg — multiple sinks

The same topic can fan out to many systems. Just change the sink type.

SinkSemanticsNotesApache Icebergexactly-onceappend or equality-delete upsertJDBC (PostgreSQL, …)exactly-oncetransactional (driver bundled / added)Kafkaat-least-onceto another topic/clusterElasticsearchat-least-oncebulk indexHTTP / Console / NeorunBaseat-least-oncewebhook / debug / REST·JDBC

2.5. Richer jobs with the SDK

For transforms or multiple sinks, use the lightweight Java SDK (no Arrow/broker deps on the client — just the JDK HttpClient + Jackson).

ItdaStreamSession session = ItdaStreamSession.builder()
    .adminUrl("http://broker:8082")
    .userToken("ITOK...")            // IAM user token (the recommended SDK/CI credential)
    .build();

String jobId = session.streamSource(Source.kafka("events").format("json"))
    .filter("event_type = 'purchase'")
    .select("id", "name", "amount")
    .map("com.example.EnrichFn")     // 1->1 user transform (on the broker classpath)
    .sink(Sink.iceberg("prod-iceberg", "analytics.purchases").upsertKeys("id"))
    .parallelism(4)
    .commitInterval(5_000)
    .start();

For authentication, prefer the user token (ITOK...) — one of the three credentials IAM issues. Creating an access key in IAM returns an access key / secret key / user token together; the user token is used via the Authorization: Token <token> header for SDK/CI auth.

3. Wrap-up

ItdaStream does “receive over Kafka, retain cheaply and indefinitely on S3, and land into Iceberg exactly once” in a single system. The basics are no-code topic-to-Iceberg; richer processing is a multi-sink pipeline via the SDK — all without a separate stream-processing cluster.

• Docs: https://cloudcheflabs.github.io/itdastream-docs

Kafka API Compatible · S3 Tiered Storage · Exactly-Once Streaming to Iceberg.