Python Shared Memory + PyArrow IPC: Zero-Copy Dataframes Across Workers

Share one Arrow-backed dataframe across multiple Python processes without pickling storms — by writing Arrow IPC once into shared memory and letting every worker read it zero-copy.

The first time you profile a “simple” multiprocessing pipeline and see the same dataframe copied eight times, it’s a little heartbreaking. You didn’t even do anything fancy. You just… passed data to workers. Suddenly your memory graph looks like a staircase, your cache is thrashing, and latency gets weirdly spiky.

Let’s be real: in Python, the enemy isn’t always compute. It’s copies.

So here’s a pattern that feels like cheating in the best way:

Put an Arrow IPC payload into multiprocessing.shared_memory, then open it in each worker using PyArrow’s zero-copy IPC readers.

One producer. Many consumers. Minimal duplication.

Why “just pickle it” quietly ruins throughput

When you pass a Pandas dataframe through a multiprocessing.Queue or submit it to a ProcessPoolExecutor, Python typically serializes (pickles) it. That often means:

• converting to a byte stream
• copying buffers into that stream
• copying again into the child process memory space
• reconstructing objects on the other side

Even when libraries optimize parts of this, the general shape remains: rows become bytes, bytes become new memory.

Arrow flips the model. Arrow is already a standardized, columnar memory layout. IPC is Arrow’s native way to ship those columns around. And importantly: IPC reads can be zero-copy when the input source supports it (like a memory map or a pyarrow.BufferReader).

Shared memory gives you the transport. IPC gives you the layout. Together, they give you the “why are we copying this?” moment.

The two primitives you’re composing

1) Python shared memory: a named byte buffer across processes

Python’s multiprocessing.shared_memory module provides a SharedMemory class for allocating and managing shared memory blocks accessible by multiple processes.

It’s intentionally low-level. Which is both power and responsibility:

• you must manage lifecycle (close(), unlink())
• you must avoid concurrent writes (or use locks)
• you must not “forget” segments (they can linger if not unlinked)

2) Arrow IPC: a wire format that’s already in-memory friendly

Arrow IPC supports a streaming format and a file format for serializing record batches and tables. And Arrow’s docs are blunt about the benefit: IPC is optimized for zero-copy patterns, and reads can stay zero-copy if the source allows it.

One caveat: IPC is “predominantly zero-copy,” but certain features (like compression) can force allocations during read.

The architecture: one writer, many readers

Here’s the shape:

(create Arrow table / record batches)
Producer process
    |
    |  Arrow IPC stream bytes
    v
SharedMemory segment  [name="df_shm_42", size=N]
    |
    +--> Worker 1 attaches by name -> BufferReader -> ipc.open_stream() -> Table
    +--> Worker 2 attaches by name -> BufferReader -> ipc.open_stream() -> Table
    +--> Worker 3 attaches by name -> BufferReader -> ipc.open_stream() -> Table

The only “copy you can’t avoid” is writing the IPC bytes into the shared memory segment once. After that, every worker reads from the same underlying bytes.

Working code: stream an Arrow table into shared memory and open it zero-copy

This example uses Arrow’s streaming IPC. It’s simple and perfect for “broadcast this dataset to N workers.”

Producer: serialize once, publish name + size

import pyarrow as pa
import pyarrow.ipc as ipc
from multiprocessing import shared_memory

def publish_table_to_shm(table: pa.Table) -> tuple[str, int]:
    # 1) Serialize to IPC bytes in-memory
    sink = pa.BufferOutputStream()  # resizable output stream :contentReference[oaicite:5]{index=5}
    with ipc.new_stream(sink, table.schema) as writer:
        writer.write_table(table)
    buf = sink.getvalue()  # pa.Buffer containing IPC stream bytes

    # 2) Allocate shared memory and copy bytes once
    shm = shared_memory.SharedMemory(create=True, size=len(buf))
    shm.buf[: len(buf)] = buf.to_pybytes()  # one copy into shared memory

    # IMPORTANT: producer keeps shm alive as long as consumers need it.
    return shm.name, len(buf)

Worker: attach and open stream without copying buffers again

import pyarrow as pa
import pyarrow.ipc as ipc
from multiprocessing import shared_memory

def load_table_from_shm(name: str, nbytes: int) -> pa.Table:
    shm = shared_memory.SharedMemory(name=name, create=False)

    # Wrap shared memory as an Arrow buffer (zero-copy view over bytes)
    view = shm.buf[:nbytes]                 # memoryview
    arrow_buf = pa.py_buffer(view)          # Arrow buffer from Python buffer protocol :contentReference[oaicite:6]{index=6}
    reader = pa.BufferReader(arrow_buf)     # zero-copy reader :contentReference[oaicite:7]{index=7}

    stream = ipc.open_stream(reader)
    table = stream.read_all()

    shm.close()  # do NOT unlink in workers; producer controls lifecycle
    return table

A few notes that make this production-friendly:

• Arrow IPC streaming writers/readers are the intended API (new_stream, open_stream).
• BufferReader is explicitly designed for zero-copy reads from buffer-like objects.

Yes, the producer example uses buf.to_pybytes() which copies into a Python bytes object before writing into shared memory. That’s fine as a baseline, but if you want to be more serious about minimizing intermediate copies…

The “no extra buffer” upgrade: write directly into a preallocated Arrow buffer

PyArrow provides a FixedSizeBufferWriter, a stream that writes into an already allocated pyarrow.Buffer.

In theory, the cleanest approach is:

1. Allocate shared memory of size N
2. Wrap it as an Arrow-writable buffer
3. Write IPC bytes directly into it

In practice, you still need one hard thing: knowing N up front. Most teams solve this with a two-phase approach:

• write to BufferOutputStream() to compute len(buf)
• allocate shared memory
• then copy once (as shown above)

It’s “one extra copy” but still a massive win compared to N copies per worker.

Choosing IPC stream vs IPC file

Arrow IPC supports both:

• Streaming format: sequential batches, great for “broadcast once, read all.”
• File format: includes a footer and is better for random access, but you’re still sharing bytes the same way.

If workers only need a subset of batches, file format can be attractive. If they need “the whole dataframe,” streaming is simpler.

Also remember: enabling compression can introduce allocations during IPC reads, so leave it off unless bandwidth is your bottleneck.

Real-world use case: CPU-bound feature generation with a shared base table

Picture an ML scoring service:

• One request fetches a 500MB feature table (Arrow table)
• Eight worker processes run independent transforms (different models / different feature slices)
• You want parallelism without loading that 500MB eight times

Shared memory + IPC gives you a neat split:

• one process fetches/constructs the base Arrow table
• it publishes shared memory name + size
• workers attach and compute without duplicating the base dataset

The compute scales. Memory doesn’t explode.

Guardrails you should not skip

1) Treat shared memory as immutable to consumers

One writer, many readers. If multiple processes write, you need synchronization and versioning.

2) Make lifecycle explicit

Python shared memory is powerful because it’s shared… and dangerous because it can linger.

• workers: close()
• producer: close() and unlink() when all consumers are done

The shared_memory docs call out lifecycle management and provide helpers like SharedMemoryManager for coordinating this across processes.

3) This is same-host only

Shared memory doesn’t cross machines. If you want zero-copy-ish transport over the network, look at Arrow Flight / Flight SQL instead (different pattern, different tradeoffs).

A quick aside: why not Pickle protocol 5 out-of-band buffers?

It’s a legit alternative.

PEP 574 (pickle protocol 5) adds out-of-band buffers so large data can be handled separately from pickle metadata, enabling zero-copy handling in some cases.

If your pipeline already leans on pickle semantics, protocol 5 can be a pragmatic upgrade. But Arrow IPC has two big advantages for dataframes:

• it’s a language-neutral columnar layout
• it plays cleanly with Polars, DuckDB, Spark connectors, and more

So: pickle5 is a good tool. IPC is a better “dataframe interchange” tool.

Conclusion: stop copying your biggest object

If you’re doing multiprocess analytics, feature engineering, or batch transforms, the “default” approach often pays a hidden tax: redundant memory and redundant serialization.

Python shared memory gives you a shared byte slab. PyArrow IPC gives you a format that can be read zero-copy when backed by buffer-friendly sources like BufferReader.

Put them together and you get a pattern that’s simple, fast, and oddly satisfying: one dataset, many workers, no memory staircase.

If you try it, comment with your workload (Pandas → Arrow? Polars? pure Arrow?), dataset size, and number of workers. And if you want a follow-up, I can show a “production” version with:

• shared memory reference counting
• a small header for schema + length
• and an optional fallback to mmap files for very large payloads