The Past and Present of Stream Processing (Part 26): The Ark Built by Independent Developers — ArkFlow

ArkFlow, released in early 2025, is an experimental Rust-based stream processing engine with a unique feature: deep integration with AI/ML through native Python support. Built on Tokio’s async runtime and Apache Arrow, the current version is 0.4.0-rc1, with approximately 1.2k stars on GitHub, and explicitly warns against production use.

While the project demonstrates considerable technical maturity — combining Rust’s high performance with Python’s machine learning ecosystem through PyArrow’s zero-copy data exchange mechanism and DataFusion’s query engine — it also has critical limitations, including:

• Fundamentally stateless architecture
• Lack of exactly-once semantics support
• Almost no real production use cases

These limitations make ArkFlow more of an early technical experiment rather than a production-grade solution that can replace mature engines like Apache Flink or Kafka Streams.

About the Author

The project’s unique selling point lies in its seamless real-time AI inference capabilities, but this differentiation currently remains largely theoretical — there are no public performance benchmarks, production cases, or state management mechanisms required for complex stream processing.

ArkFlow’s author is Chen Quan, a developer from Chengdu who is active in open-source project development. He released version 0.1.0 on crates.io around March 2025. The project was initially hosted under his personal GitHub account chenquan/arkflow, then migrated to the organization-level repository arkflow-rs/arkflow to support broader community development. During this organizational transition, Chen Quan remains the project’s primary maintainer, responsible for most merge requests and version releases.

Despite the young codebase, its development pace is remarkably rapid. Version 0.3.0, released in May 2025, was the first major feature release, introducing WebSocket, NATS, and Redis input sources, as well as windowing aggregation capabilities (session windows, sliding windows, tumbling windows). This version also added VRL (Vector Remap Language) processor support (used in Vector) and experimental distributed computing functionality.

By June 2025, the project received its most strategically significant update: Python processor support through PyO3. The official blog post “ArkFlow+Python: Easy Real-time AI” announced this feature, emphasizing that the engine bridges the gap between performance and the AI ecosystem.

Version 0.4.0-rc1 further expanded the ecosystem, adding Industrial IoT support (Modbus input), database outputs (MySQL and PostgreSQL), and multi-cloud object storage integration (AWS S3, Google Cloud Storage, Azure Blob Storage, HDFS).

ArkFlow has been listed in the CNCF Cloud Native Landscape and established a Discord community server. However, currently only one production user is confirmed (Conalog in South Korea), and GitHub activity remains lower compared to competitors. The coexistence of the personal repository (chenquan/arkflow, 243 stars) and organizational repository (arkflow-rs/arkflow) also reflects that the early-stage organizational structure has not fully stabilized.

Core Problem

ArkFlow aims to solve a core problem: how to deploy machine learning models on streaming data without requiring complex integration layers or systems programming expertise. Traditional stream processing architectures often require data scientists to deeply learn Java/Scala/Rust or use complex integration mechanisms, which introduce latency and operational overhead. The gap between model development in Python notebooks and real-time deployment in production stream processing systems has been a major barrier to implementing AI in time-sensitive applications. ArkFlow’s configuration-driven design and Python integration are particularly suitable for data scientists and machine learning engineers who want to deploy real-time models without deeply understanding the complexities of stream processing systems.

However, its stateless architecture has fundamental limitations. Scenarios requiring stateful computation — such as complex session aggregation, multi-stream joins over long time windows, or maintaining feature stores to enrich model inputs — are currently beyond ArkFlow’s capabilities. The official documentation explicitly states:

“Currently ArkFlow is stateless, but it can still help you solve most data engineering problems.”

However, this “most” means it’s not suitable for many complex streaming application scenarios — precisely where mature engines like Flink and Kafka Streams excel.

Configuration Model

ArkFlow adopts a declarative YAML configuration model, very similar to Benthos (as well as ByteWax and Pathway), defining streaming pipelines through structured specifications rather than imperative code. Each stream configuration forms a standard processing pipeline: input → [buffer] → pipeline → output + error_output. This pattern achieves separation of concerns—data sources, transformation logic, backpressure control, and destination routing each reside in independent configuration sections.

A minimal runnable example:

yaml

logging:
  level: info

streams:
  - input:
      type: "generate"
      context: '{ "timestamp": 1625000000000, "value": 10, "sensor": "temp_sensor" }'
      interval: 1s
      batch_size: 10

    buffer:
      type: "memory"
      capacity: 10
      timeout: 10s

    pipeline:
      thread_num: 4
      processors:
        - type: "json_to_arrow"
        - type: "sql"
          query: "SELECT * FROM flow WHERE value >= 10"

    output:
      type: "stdout"

    error_output:
      type: "stdout"

In production environments, Kafka-to-Kafka processing is the most common pattern. The configuration file defines consumer group, broker address, topic, and client ID. The pipeline executes a series of transformation steps on messages — parsing from JSON to Apache Arrow format, then filtering or aggregating through SQL, and finally serializing for output. A separate error_output path routes failed messages to dedicated error topics for subsequent analysis or replay.

ArkFlow’s Python processor supports embedded scripts or external module imports for custom logic and machine learning model inference. The configuration specifies function name, module path, and Python dependency path. The processing function receives PyArrow’s RecordBatch and returns a list of processed batches, for example:

def process_batch(batch):
    import pyarrow.compute as pc
    value_array = batch.column('value')
    doubled = pc.multiply(value_array, 2)
    # Return modified batch
    return [batch]

The learning curve is moderate. Developers with some understanding of the Rust toolchain, YAML configuration, stream processing concepts, or messaging systems can get started relatively quickly. The configuration structure is logically clear, and examples in the official documentation provide a good starting point. However, challenges arise with deeper use:

• Understanding Apache Arrow’s columnar storage format for complex data processing
• Mastering processor-specific query languages (such as SQL, VRL)
• Debugging without mature operational tools
• Constraints imposed by the current stateless architecture

Common workflow patterns include:

• Real-time filtering: Read data from Kafka topics, apply SQL WHERE conditions, write to output topics
• Stream aggregation: Use windowed buffers to collect messages over time intervals, then perform GROUP BY operations
• Data transformation: Implement parsing, schema transformation, and encoding chains through multiple processors
• AI/ML inference: Route data through Python processors to load models and perform predictions
• Multi-source aggregation: Define multiple independent stream configurations and converge results in a common output

This configuration-driven design makes ArkFlow powerful for rapid prototyping and experimental real-time AI pipeline development, but complexity increases as functionality expands.

Architecture

ArkFlow adopts a modular pipeline architecture consisting of four main layers:

1. Stream Configuration Layer

Uses declarative YAML configuration, supports multiple concurrent streams, each defined as: input → buffer → pipeline → output + error paths

2. Component Layer

Provides pluggable modules including input, buffer, processor, and output, with all components implementing common trait interfaces.

3. Runtime Layer

Based on Tokio async executor with configurable thread pools for CPU-intensive tasks and event-driven non-blocking I/O operations.

4. Data Format Layer

Uses Apache Arrow as internal data representation, implementing zero-copy operations and columnar memory layout to improve computation and cache efficiency.

Data Flow

Data flows through the system as follows:

• Input stage: Input components asynchronously read raw data and convert it to internal MessageBatch format, containing metadata such as timestamps, offsets, and data sources.
• Buffer stage (optional): Implements backpressure control through bounded queues and window-based (session/sliding/tumbling) grouping.
• Processing stage: Converts MessageBatch to Arrow RecordBatch, sequentially executes multiple processors, each receiving and outputting Arrow-format data; the system automatically handles schema evolution.
• Output stage: Converts RecordBatch to output format, writes in parallel through Tokio tasks, failed messages are routed through independent error paths.

ArkFlow adopts a micro-batch architecture, processing small batches of data to achieve low latency (rather than single messages or large batch jobs).

Batch size is configurable (typically 10 to 10,000+ messages) to balance throughput and latency. Batch boundaries don’t affect computational correctness; developers can tune based on workload.

This approach fully leverages Arrow’s columnar storage characteristics for efficient vectorized operations while maintaining sub-second processing latency.

Backpressure & Windowing

ArkFlow’s backpressure signals propagate across multiple levels:

• Buffer layer backpressure: When buffer is full, sends signal upstream, input component pauses reading through async wait
• Pipeline backpressure: When processors run slowly causing batch accumulation or thread pool saturation, triggers upstream pause through Tokio channel capacity control
• Output backpressure: When downstream output (sink) writes too slowly, propagates upstream through bounded retry queues, failed writes trigger error output paths

The system uses Tokio’s Semaphore and channel bounds to control flow rate, sending pause/resume notifications through callback mechanisms; input offsets are preserved during pause to prevent data loss.

ArkFlow implements three window types:

• Session Window Uses HashMap<session_id → window_state> to track last message timestamp for each session Window starts on first message, subsequent messages within timeout extend the window, automatically closes after timeout via timer
• Sliding Window Uses ring buffer to retain active messages, supports multiple overlapping windows Based on timestamp bucketing with efficient expired data removal
• Tumbling Window Only maintains one active window at a time, batch outputs and clears state at window boundary Lower memory footprint compared to sliding windows

Transaction Semantics

ArkFlow implements at-least-once semantics, relying on external systems’ idempotence guarantees.

For Kafka input:

• Consumer group commits offset after successful output write
• If write fails, re-consumes from last committed offset, ensuring no message loss

Input messages are acknowledged only after receiving confirmation from the sink during output stage.

The system’s dual-output architecture routes failed messages along with error metadata to error paths, supporting dead-letter queue patterns.

However, due to stateless design, the system doesn’t support exactly-once semantics:

• Duplicate processing may occur during failure recovery
• Cross-stream transactions are not supported

Therefore, ArkFlow’s transaction model is closer to a lightweight, low-latency “best-effort” mode rather than strong consistency guarantees.

Like the previously introduced Arroyo, ArkFlow is also built on Arrow and DataFusion.

Summary

ArkFlow, as an emerging open-source stream processing project, has gained considerable community recognition. Its product form is similar to Benthos, Arroyo, ByteWax, and Pathway, providing a descriptive stream processing pipeline that leverages Rust for high-performance stream processing and Python for AI and machine learning integration. These all reflect modern stream processing architectural design.

ArkFlow’s biggest weakness is its explicit statement “not suitable for production environments.” The early codebase had a warning: “‼️ Not yet ready for production, please do not use in production environments ‼️”. Although this warning has been removed from current documentation, to date there are no public case studies, corporate blogs, or conference presentations. This means the engine’s behavior in real production environments — including scaling patterns, failure scenarios, performance characteristics, and operational challenges — remains almost completely unknown.

The stateless architecture is a fundamental limitation for many stream processing applications. The official documentation acknowledges this, with the key being the “most” — it excludes applications requiring state, such as:

• Stateful aggregation maintaining running totals or session state
• Cross-stream joins requiring long-term state preservation
• Feature stores for machine learning feature enrichment
• Tasks relying on checkpoint/recovery mechanisms for exactly-once processing

In contrast, competitors like Flink, Kafka Streams, Arroyo, and Fluvio all provide mature state management. ArkFlow’s roadmap promises “will support transactions and state management features” but provides no timeline, leaving this gap long-standing.

ArkFlow currently only provides at-least-once delivery semantics, struggling to meet modern application needs. Duplicate processing may occur during failures or recovery. The system has no checkpoint or savepoint mechanisms, cannot recover from specific states, and doesn’t support cross-stream transactions. For scenarios sensitive to duplicate operations like financial payments, such limitations are almost unacceptable. Due to the inherently stateless architecture, implementing exactly-once semantics would require nearly rewriting the core.

The lack of performance benchmarks is also a serious trust issue. While documentation claims “excellent performance, low latency,” it provides no data or validation. In contrast, Arroyo explicitly demonstrates testing methodology showing “5x faster than Flink,” and Fluvio publishes specific memory usage metrics. While ArkFlow theoretically can achieve high performance relying on the Rust + Tokio + Arrow combination, without actual data these claims lack credibility.

As a project also led by an individual, ArkFlow is very similar to Benthos in many ways. It can be seen as a Rust version of Benthos. I’m pleased to see the development of stream computing open-source projects with participation from Chinese independent developers. Best wishes to this project.

Thanks for reading! I’m the co-founder and CTO of Timeplus. Proton https://github.com/timeplus-io/proton is our open-source, high-performance streaming SQL engine built in C++. It enables you to process real-time data streams using standard SQL, perfect for real-time security and monitoring, real-time machine learning pipelines, IoT analytics, and more.

Feel free to star us on GitHub or join our Slack community at https://timeplus.com/slack to discuss the future of streaming data together!

Originally Published in: Wen Shu Qi Wu (闻数起舞)