YaFF Goes Open Source: Why We Built a Zero-Copy Representation for Protobuf

YaFF — Yet Another Flat Format

Reading serialized data is an infrastructure tax that every service pays when receiving information from external sources — over the network or from disk. In the industry, Protobuf has become the de facto standard for schematized data, and this tax most often manifests as substantial CPU cost from parsing. In advanced cases, teams try to replace parsing with the considerably cheaper — but far less ergonomic — zero-copy representation offered by FlatBuffers.

We have open-sourced YaFF (Yet Another Flat Format) — a format designed to eliminate the parsing tax from hot read paths without forcing you to abandon Protobuf. At Yandex’s scale, this matters especially because changing something as foundational as a serialization format is expensive and painful. YaFF was designed from the ground up as an alternative wire format for existing Protobuf ecosystems (and, looking ahead, for FlatBuffers as well). This makes it inexpensive to adopt in existing projects without rewriting tens of thousands of lines of code.

We will illustrate how this works in practice using Yandex Ads as an example: in a recommendation system where each of hundreds of thousands of requests processes tens of thousands of objects, data representation becomes a critical concern. Thanks to YaFF, we were able to optimize the system gradually, step by step, saving 10–20% CPU in large-scale runtimes without costly refactors.

In this article, we:

• Discuss the serialization tax and explain why combining Protobuf with FlatBuffers breaks down at large scale;
• Dive deep into the internals of FlatBuffers and uncover its fundamental trade-offs;
• Study how YaFF is structured internally, how its bytes are laid out, and what makes an efficient Protobuf representation possible;
• Discuss how a unified object model simplifies life for product engineers and ML practitioners;
• Cover the open-sourcing of YaFF and how to start experimenting with it.

One caveat up front: at our scale, schemaless formats like JSON are off the table due to their verbosity and lack of strict contracts. This article deals exclusively with binary schematized formats.

The Typical Data Flow Path in a System

In modern high-load backend architectures, data passes through three distinct stages, each imposing its own — often mutually exclusive — requirements on the serialization format.

The first stage is the document-oriented store. This is the system’s global state and the source of truth for the canonical data model. The workload here is fundamentally OLTP: point updates or small-batch writes. The format at this level must be compact and support flexible schema evolution. Business entities inevitably multiply to hundreds of related structures with thousands of fields, and the format must handle that natively.

The second stage is the monolithic or microservice runtime, where product logic executes. Unlike the store, runtimes operate under tight latency constraints and compete for maximum throughput. The data format here must deliver high read speed. Memory allocation overhead or data unpacking costs become critical, while developers still need a safe and ergonomic API.

The third stage is analytics and ML pipelines. Enormous volumes of logs flow in from runtimes for offline processing, model retraining, and statistics gathering. Random access to data is not required here. The primary metrics at this stage are maximum compression ratio and fast sequential scan throughput over large datasets.

In a project’s early days, Protobuf’s generality is sufficient to cover all three stages. But as both load and data volume grow significantly, that generality becomes too expensive: the serialization and deserialization tax starts consuming double-digit percentages of CPU, translating into tens of thousands of physical cores.

At that point, the architecture inevitably stratifies toward specialized solutions: the store stays on Protobuf for the data model; analytics migrates to columnar formats (Parquet, Arrow); and performance-critical runtimes demand zero-copy tools — such as FlatBuffers or even raw C++ structs.

The Field Conversion Problem

Although specialized formats solve performance problems within their respective stages, at the boundaries between systems, they generate a new organizational tax: the need for constant data translation.

Consider a typical scenario: an ML engineer or product developer wants to add a feature. When the format is unified, this is straightforward. But with a zoo of formats, adding a single field turns into a cross-team quest:

• Add the field to the Protobuf description of the data model in the store;
• Coordinate with the data-preparation team to propagate the field to the runtimes;
• Sync with data engineers to ensure the field correctly lands in the logs.

The result is an architecture wrapped in layers of converters, adapters, and custom translation logic. Every such layer is a potential source of bugs, requires its own testing, and consumes maintenance resources. The complexity of moving data through all these layers increases time-to-market for new features and slows down product experiments.

In an ideal world, you would want to combine the incompatible: bring back the simplicity of the project’s early stages — where a single Protobuf schema served as the source of truth for the entire system — while still getting the high performance of specialized formats under the hood. That architectural challenge is exactly what led us to build YaFF.

In this article, we focus on the hottest boundary: the transition of data from the store to the runtime. Since all performance-critical runtimes at Yandex are written in C++, the discussion below stays in that context.

Can We Really Not Get By Without Protobuf?

We won’t dwell at length on attempts to squeeze every last drop out of standard Protobuf. Yes, using arenas reduces heap fragmentation, and the string_view API eliminates unnecessary allocations when copying strings. In the early stages of scaling, this genuinely helps. But it doesn’t resolve the format’s fundamental problem: the data still has to be parsed.

To understand where Protobuf’s applicability ends, let’s look at some numbers. One of the high-throughput Ads services processes roughly 4,000 RPS per host, and for each request, it must evaluate several hundred banners. That gives us millions of objects per second on a single machine. Moving such volumes over the network is simply not feasible, so the entire shard — 30 million banners weighing around 50 GB — must reside locally.

If these data are stored as Protobuf, there are two equally bad options:

• Parse on the fly: deserializing a million objects per second will burn through enormous amounts of CPU.
• Parse at startup: you could parse 50 GB into C++ structs during startup. But that means replica startup takes tens of minutes (particularly bad during incidents) and consumes a large amount of RAM.

In performance-critical systems, the only viable pattern for volumes of this scale is memory-mapped files. We place the file on disk (or in tmpfs) and transparently map it into virtual memory. But for this to work, the format must itself be mappable: the bytes on disk must lie in exactly the form the processor expects to see them in memory.

Even at loads an order of magnitude smaller — where data can still be moved over the network — a mappable format offers huge benefits. It frees CPU, and a local response cache automatically becomes a mappable index that survives service restarts without a slow warmup.

The ability of a format to allow data to be read directly, without intermediate copying or parsing, is the foundation of the zero-copy paradigm. This is where its most well-known C++ representative enters the scene: FlatBuffers.

FlatBuffers — Just Protobuf Without Deserialization?

In the community, FlatBuffers is often perceived as essentially Protobuf without deserialization. The illusion of interchangeability is reinforced by their common origin (both formats were created at Google), similar schema syntax, type sets, and tooling.

Even the official FlatBuffers documentation openly states their similarity:

“Protocol Buffers is indeed relatively similar to FlatBuffers, with the primary difference being that FlatBuffers does not need a parsing/unpacking step to a secondary representation before you can access data, often coupled with per-object memory allocation.”

Moreover, if you look at the flatc compiler’s help text, you’ll find a — proto flag that promises automatic translation of .proto files into .fbs:

“ — proto: Expect input files to be .proto files (protocol buffers). Output the corresponding .fbs file…”

It seems like the repeated conversion problem is solved: take your canonical Protobuf schemas from the store, feed them to the FlatBuffers compiler, and get a zero-copy runtime.

In practice, however, any attempt at such a seamless migration turns painful. Despite their surface similarity, they are semantically very different, and simply layering FlatBuffers on top of an established Protobuf model won’t work. We’ll get to where the differences come from shortly. For now, let’s walk through the most significant problems you’ll run into as a user if you try to use FlatBuffers as a drop-in replacement for Protobuf.

FlatBuffers Is NOT Protobuf Without Deserialization!

Schemas

The differences start at the schema level. A schema defines a long-term contract in which the format specifies not just the current structure, but also the rules for its evolution. The — proto flag lets you do a one-shot conversion of a .proto file into a .fbs. But keeping them in sync going forward is not feasible.

FlatBuffers’ schema evolution rules are stricter. New fields may only be added at the end of a message. Deleting fields from the schema is forbidden; they can only be marked deprecated. In Protobuf, by contrast, you’re free to delete fields from the middle, safely reserving their identifiers with the reserved keyword.

Let’s see what this semantic mismatch looks like in practice.

Original Protobuf (user.proto)

message User {
  reserved 2; // Deleted Field
  string email = 3;
  int32 id = 1;
}

Result of flatc –proto (user.fbs)

table User {
  email: string; // Implicit id: 0
  id: int;       // Implicit id: 1
}

What happened here? The flatc compiler ignored reserved and discarded the original Protobuf field tags, assigning identifiers simply by field order. As a result, it becomes impossible to build a compatible automatic converter: the critical metadata about schema evolution is irretrievably lost.

And this is just one of many compatibility issues. If you try to maintain a unified data model, you will encounter a whole class of similar problems at the schema level.

User-Facing API

The schema is only half the contract. The other, equally important, part is the generated C++ API that developers interact with every day. And here the gap between the Protobuf and FlatBuffers ecosystems becomes even more apparent.

In large systems, data models inevitably become deeply nested. In the Protobuf world, developers are accustomed to safe accessor chains. If an intermediate object doesn’t exist, Protobuf transparently returns a reference to a default instance. FlatBuffers provides a lower-level API: any missing nested object is returned as a null pointer.

When data is heavily sparse — where different product slices populate different subtrees of the schema — you cannot skip null checks and rely on verbal agreements. Missing even a single check will eventually lead to a null pointer dereference at runtime.

The result is that business logic drowns in infrastructure boilerplate. The code becomes brittle, hard to read, and expensive to review.

Using Protobuf

uint32_t total_score = req.user().stats().score() + req.context().device().score();

Using FlatBuffers

uint32_t total_score = 0;

if (auto user = req->user()) {
    if (auto stats = user->stats()) {
        total_score += stats->score();
    }
}

if (auto context = req->context()) {
    if (auto device = context->device()) {
        total_score += device->score();
    }
}

The Conversion Problem Persists

Even if you’re willing to ignore the semantic schema conflicts and accept the boilerplate on reads, another source of problems awaits you: serialization. The standard flatc compiler can (with caveats) translate schemas, but it does not generate code to convert data from a Protobuf object into a FlatBuffers buffer.

This means that manual field conversion will never go away, and you’ll have to write and maintain converters yourself.

The Beginning of Our Journey and Its Difficulties

All of the problems described above live in the realm of interfaces and contracts, which means they can be worked around with additional layers of abstraction.

In our first iteration, we went exactly that route: writing our own code-generation layer on top of FlatBuffers to impose Protobuf semantics on it. With hacks and workarounds, this approach worked — imperfectly, but it worked:

• we handled the reserved field problem by generating fake dummy fields — since old data is never written into them, binary compatibility is preserved (albeit using an undocumented capability);
• we addressed the fragile API problem by generating smart C++ wrappers that hid all null-pointer checks under the hood and transparently returned default values;
• we solved the manual serialization problem by auto-generating mappers that built the FlatBuffers buffer from a Protobuf object in the correct field order.

Our initial approach to the problem is a story that warrants its own article, but there's not enough space to explore it in detail here.

What About Performance?

Having resolved the ergonomics problems at the compiler level, we hit a limitation that code generation cannot overcome: performance.

There are broadly two data-access profiles in runtimes:

Network I/O: data arrives over the network (objects from external stores or responses from external services), and the main goal is to eliminate the CPU tax of deserialization.

Local index reads: data lives in large mmap indexes, and the main goal is to squeeze out maximum throughput under tight latency constraints.

In services that primarily read indexes from memory (as in the previous example), the bottleneck is often not the processor but the memory bus bandwidth. Cores sit idle waiting for data from RAM, manifesting as a drop in IPC (Instructions Per Cycle).

For the first class of tasks, FlatBuffers is usually adequate, but under index-heavy load its overhead becomes too pronounced. To understand why, let’s look under the hood of the format using a benchmark with hierarchical data.

Simulating Production Load

To objectively measure the overhead we need a synthetic test that accurately approximates the access patterns of real business logic. Our repository contains a full suite of different benchmarks, but for this article we chose the most illustrative one: a test for reading deeply nested hierarchies.

We use the following schema.

message Leaf {
   optional uint64 a = 1;
   // Field 2..9
}

message Intermediate {
   // Field 1..8
   optional Leaf leaf = 9;
}

message Root {
   // Field 1..8
   optional Intermediate intermediate = 9;
};

The benchmark payload is deliberately simple: we descend the object tree and sum the scalar fields in the deepest leaf.

sum += root->intermediate()->leaf()->a();
// Field 2..8
sum += root->intermediate()->leaf()->i();

Despite its apparent simplicity, this test measures one of the core aspects of any serialization format — the efficiency of representing composite data structures (message in Protobuf, table in FlatBuffers, struct in C++).

As our baseline we use raw C++ structs. There are two ways to physically model the hierarchy: inline nested objects or store them by pointer. In this article we focus on the pointer approach. This gives a fairer comparison because it more accurately reflects the semantics of optional fields in both Protobuf and FlatBuffers.

There are two scenarios for working with structs:

random access, where a small number of fields are read from a large collection of structs — in this case performance is determined by the number of reads from distinct cache lines and the predictability of those reads;

local access, where a large number of fields are read from a single struct — here the format’s overhead and the efficiency of the generated code come to the fore.

Real systems encounter both scenarios, and must handle either efficiently. Our design accounted for both: full benchmarks for each access pattern are implemented and available in our repository. For the detailed analysis in this article, however, we chose local access for two reasons: it is the more principled cache-aware pattern that high-load systems strive for, and it lets us examine how all operations — not just random reads — affect performance.

We build the benchmark with Clang 20.1.8 with all optimizations, running on AMD EPYC 7713. The results for raw structs vs. FlatBuffers are as follows.

We see a 4.5× difference, which matches our observations in real runtimes. To understand where this gap comes from, we need to look under the hood of FlatBuffers — specifically, at how it represents composite types.

How FlatBuffers Works

FlatBuffers is built around the vtable concept. Every message begins with a 4-byte offset to the vtable. The vtable is a compact metadata dictionary consisting of 2-byte slots: the first slot holds the vtable’s size, followed by offsets from the start of the table to the actual data, strictly in field-id order.

This table-based approach has several advantages: it allows fields to be written in any order while still being properly aligned; it permits uninitialized fields to be skipped with the absence recorded; and it allows the schema to evolve by appending new fields at the end. Meanwhile, the offset at the start of a message enables vtable deduplication: two messages with identical layout can point to the same vtable.

The cost, however, is a higher metadata volume and a more complex read algorithm. To retrieve the value of a single field we need to:

1. Read the start offset and jump to the vtable.

2. Read the vtable size and check whether our field falls within its bounds. This is where backward-compatibility magic happens: if the requested field id lies beyond the vtable’s size, the new code is reading an old message and can return a default.

3. Read the 2-byte offset for the requested field from its slot and check for zero. A zero value means the field is uninitialized — return the default again.

4. Finally, return to the start of the message and read the actual data: a scalar or another offset to a nested object. For a nested object, one more indirection is added.

This is a fairly elegant algorithm on paper, but it makes it clear why it incurs such a large performance penalty compared to C++ structs.

First, there is pure overhead from instruction count. Accessing a field in a C++ struct compiles to a single memory read at a constant offset. Here, to reach the actual payload, we perform four reads, two branches, and arithmetic.

Second, this algorithm maps poorly to the processor’s instruction pipeline. All four reads are dependent: to read the data in the second step, you must first wait for the result from the first step, and so on. Furthermore, for large messages, at least two — and often all four — of these reads may fall on different cache lines. The dependent read chain leaves the processor little room to overlap these operations, and what you are effectively reading is random pieces of memory that continuously evict useful cache lines.

The Compiler Doesn’t Understand FlatBuffers Well

Beyond the chained reads from different parts of memory that stall the instruction pipeline, working with FlatBuffers also creates difficulties at the compiler level for TBAA (Type-Based Alias Analysis).

To verify this, we can compare the performance of two summation styles.

Direct Summation

s += root->intermediate()->leaf()->a();
s += root->intermediate()->leaf()->b();
// ...
s += root->intermediate()->leaf()->i();

Manual Caching

const auto* l = root->intermediate()->leaf();
s += l->a();
s += l->b();
// ...
s += l->i();

Intuitively, the two styles above should perform about the same, and the choice would seem to be purely a matter of taste. In reality, the benchmark difference is nearly 2×.

This happens because in the standard C++ FlatBuffers implementation, any composite message is implemented via something resembling a Flexible Array Member, and reads ultimately reduce to a reinterpret_cast to a pointer of the desired type. Put simply, the generated code does the following.

Table Type Declaration

struct I {
    // Field accessors;
private:
    // ...
    uint8_t data_[1];
}

Code using FlatBuffers

struct O {
    // ...
    const I* GetInner() const {
        // Calculates pointer p based on data_;
        return reinterpret_cast<const I*>(p);
    }
    // ...
}

Because of this type-punning approach, TBAA algorithms cannot give the optimizer sufficiently strong guarantees, and LLVM’s Alias Analysis frequently returns the conservative verdict MayAlias. Combined with branches, this severely limits the optimizer. CSE/GVN mechanisms cannot legally reuse computed addresses of nested structures across accesses because there is no mathematical proof that the memory is unchanged across conditional jumps.

In other words, the heavy use of accessor chains — perfectly normal in the Protobuf world — significantly degrades performance, because an already long field-read sequence is multiplied by each level of nesting.

This behavior can be validated either by reading the generated assembly — where you’ll see a repeating pattern of nested structure reads — or by running LLVM aa-eval on the generated LLVM IR and examining the Alias Analysis answers.

Generated Assembly Code

# root->intermediate()
# resolve vtable from data_
movsxd rdx, dword ptr [rdi]
mov rax, rdi
sub rax, rdx
neg rdx

# check vtable size
movzx esi, word ptr [rax]
cmp si, 21 # jump skipped

# read offset and check
movzx   eax, word ptr [rdi + rdx + 20]
test rax, rax # jump skipped

# read offset to intermediate
lea rcx, [rdi + rax]
mov eax, dword ptr [rdi + rax]
add rax, rcx

# resolves leaf from rax and reads data
# ...
# resolves intermediate and leaf
# again and again
movzx   eax, word ptr [rdi + rdx + 20]
test rax, rax # jump skipped
# ...
movzx   eax, word ptr [rdi + rdx + 20]
test rax, rax # jump skipped

Alias Analysis Results

// txt
Function: SumFbs: 92 pointers, 0 call sites
  // The MayAlias response is used
  // whenever the two pointers might
  // refer to the same object.

  MayAlias: i16* %add.ptr.i.i.i.i.i, i32* %root
  MayAlias: i16* %add.ptr.i.i.i.i, i32* %root
  MayAlias: i32* %add.ptr.i.i.i, i32* %root
  MayAlias: i32* %add.ptr.i.i.i, i16* %add.ptr.i.i.i.i.i
  MayAlias: i32* %add.ptr.i.i.i, i16* %add.ptr.i.i.i.i
  MayAlias: i32* %cond.i.i.i, i32* %root

  // ...

  MayAlias: i16* %add.ptr.i.i.i.i.i356, i64* %add.ptr.i.i378
  MayAlias: i16* %add.ptr.i.i.i.i360, i64* %add.ptr.i.i378
  MayAlias: i32* %add.ptr.i.i.i364, i64* %add.ptr.i.i378
  MayAlias: i64* %add.ptr.i.i378, i32* %cond.i.i.i358
  MayAlias: i16* %add.ptr.i.i.i.i370, i64* %add.ptr.i.i378
  MayAlias: i16* %add.ptr.i.i.i374, i64* %add.ptr.i.i378

Why Did This Happen?

When you see the performance penalties caused by the message structure, a natural question arises: why did Google’s engineers choose such a complex and multi-layered design? The answer lies in a fundamental law of software engineering: every design is built on trade-offs.

FlatBuffers’ architecture is a deliberate choice in favor of compact serialized representation at the expense of raw read speed. Each decision addresses a specific problem:

1. Moving the vtable into a separate structure at an offset allows it to be reused for arrays of identically typed objects. If a message contains a thousand identical structs, their offsets can all point to the same region of memory.

2. The vtable slots decouple a field’s logical id from its physical address. This allows fields to be written in any order while strictly respecting memory alignment rules — which is critical for correctness on any hardware.

3. An uninitialized field simply doesn’t occupy space in the main structure. An empty slot in the vtable costs just 2 bytes, and the parser doesn’t even need to know the type of an absent field. Although the vtable structure technically allows fields to be removed from the middle of a schema, the FlatBuffers standard explicitly forbids this — even though it was precisely this loophole that allowed us to build our code-generation approach.

4. Storing the vtable’s own size provides an elegant mechanism for forward/backward compatibility when new fields are added.

The Context Problem

To understand these trade-offs you need a bit of historical context. FlatBuffers was originally developed for mobile game development. Its original trade-offs follow from that: supporting a wide range of mobile processors demands aggressive alignment, while the limited resources of mobile devices demand saving on both network bandwidth and memory.

FlatBuffers is therefore not simply “Protobuf without parsing.” It is a tool with a specific set of trade-offs, and those trade-offs are not optimized for the index-heavy server-side runtimes we are discussing here.

We realized that a tool that would truly be “zero-copy Protobuf for high-load backends” simply didn’t exist on the market. So we started designing our own.

Time to Build Yet Another Flat Format

We named the new format YaFF — Yet Another Flat Format. But the key question wasn’t the name; it was the compatibility boundary: how much of the existing ecosystem were we willing to change in exchange for zero-copy reads?

The answer turned out to be simple: we didn’t want to change anything, and we wanted to reuse the established Protobuf ecosystem as much as possible. Why? Because Protobuf is already deeply embedded in many projects: around it there are schemas, code generators, evolution rules, code reviews, tests, and a large body of business logic. Building a parallel ecosystem is possible, but migrating to it would be far too costly.

Reusing Existing Schemas

The first fundamental decision in YaFF: the format will have no schema language of its own. The source of truth remains a plain .proto file.

Our experience with FlatBuffers made it clear why a separate schema becomes a problem. The same business entity ends up with two descriptions:

• proto — for the store and the shared data contract;
• fbs — for the runtime representation.

These schemas then need to evolve in lockstep: add fields in two places, monitor compatibility, and maintain converters.

So in YaFF, there is one schema: the developer continues to describe data in Protobuf as they always have, while YaFF handles the conversion to a zero-copy representation under the hood — no retraining of developers, no duplicated work.

But if the proto schema remains the source of truth, supporting only the message structure is not enough — YaFF must also replicate Protobuf’s schema evolution model. This means that safe Protobuf migrations — such as adding a field with a free id or removing a field from the middle of the schema and marking it reserved — must remain safe in YaFF as well. Otherwise the unified contract is no longer unified: the same schema starts behaving differently in different layers of the system.

Reusing the Existing User API

We’ve settled the schema question: the source of truth remains the .proto file. The next question is how the developer interacts with the data.

Reusing schemas solves only part of the migration problem. In a large codebase, preserving the familiar data access patterns is equally important: if reading a field requires rewriting application logic, the migration can cost more than the optimization gains. The transition to YaFF must therefore be nearly seamless for code that reads data. Ideally, the programmer shouldn’t even notice that under the hood, instead of a materialized Protobuf object, there is a YaFF binary buffer.

Due to C++’s strict typing, however, it is impossible to fully preserve the same types. In zero-copy mode, YaFF must not materialize data into ordinary Protobuf classes. And to read data directly from the buffer, YaFF will inevitably need to generate its own set of classes.

In other words, given existing code written for Protobuf, we want YaFF representations to be used identically. Ideally, application code should have no idea which representation it was handed.

Code with Protobuf

uint32_t GetScore(const proto::User& u)
{
    if (!u.has_stats()) {
        return 0;
    }
    return u.stats().score();
}

Code with YaFF

uint32_t GetScore(const yaff::User& u) {
    if (!u.has_stats()) {
        return 0;
    }
    return u.stats().score();
}

Universal Code

template <class U>
uint32_t GetScore(const U& u)
{
    if (!u.has_stats()) {
        return 0;
    }
    return u.stats().score();
}

From this follow the requirements for the API that YaFF will generate:

Interface compatibility: getter names, has_ methods, enum values, and access semantics must be as close as possible to Protobuf.

Template compatibility: developers must be able to write generic code that compiles and works equally with both Protobuf and YaFF representations.

Bidirectional conversion: YaFF must be able to build a zero-copy buffer from an existing Protobuf object and, when needed, recover a regular Protobuf object from that buffer.

The first two properties lower the cost of migrating application code. The third is especially important for incremental rollout. Not all parts of a system are equally sensitive to deserialization costs. Some benefit from reading data directly from the YaFF buffer, while others — less heavily loaded services, debugging tools, or languages for which a zero-copy API hasn’t yet been implemented — may be better off continuing with ordinary Protobuf. The ability to jump from the YaFF world into the Protobuf world at any time is therefore very useful.

// Service 1: data preparation
proto::User pb = LoadUser();
auto buffer = yaff::SerializeUser(pb);

// Service 2: buffer can be obtained from an external source
const auto& user = ReadRoot<yaff::User>(buffer);
uint32_t score = GetScore(user);

// Restoring proto for transfer to service 3
proto::User restored;
user.ParseTo(restored);

Immutability of the Serialized Representation

The next fundamental decision concerns the write model: once built, a YaFF buffer is immutable. At first glance this sounds like a serious limitation. But for high-load runtime scenarios it is a good trade-off.

First, supporting mutable values doesn’t mix well with a compact zero-copy representation. As soon as strings, arrays, or variable-size nested objects appear in the schema, you’re forced to decide where to write the new value, what to do with the old memory region, and how not to break offsets to the rest of the data. This leads to data fragmentation, which increases memory consumption.

Second, in a high-load runtime, data is almost always only read: it either sits in large local indexes or arrives from external systems and flows through business logic unchanged. An immutable buffer can be safely and efficiently read from multiple threads without thinking about synchronization or cache-line invalidation. Usually, if a system is looking for zero-copy reads, it is also eager to save on synchronization primitives.

If data occasionally does need to change, that logic is better pushed to a higher level: build a new buffer, or apply a delta from another buffer at read time in the runtime. The serialized buffer itself stays simple, compact, and optimized for its primary use case: fast reads.

YaFF as an Alternative Wire Format for Protobuf

Putting the previous decisions together, the idea of YaFF can be stated in a single sentence: it is an alternative wire format for Protobuf that supports zero-copy reads.

The key insight is that we don’t change the data model. Protobuf schemas remain the source of truth, schema evolution rules remain Protobuf-compatible, and the user API closely resembles the familiar Protobuf API. Only the physical representation changes: data can be stored in a binary buffer from which the runtime reads fields directly, without a deserialization step.

This design preserves a bridge between the two worlds. A YaFF buffer can be built from an ordinary Protobuf object and, when needed, a Protobuf object can be restored from a YaFF buffer. Because of this, YaFF doesn’t require a big-bang migration of the entire system. It can be introduced as a transparent optimization in the modules where parsing and allocation genuinely become the bottleneck, while leaving other components on ordinary Protobuf.

Keeping the Basic Zero-Copy Model from FlatBuffers

FlatBuffers doesn’t work for us as an off-the-shelf ecosystem, but its basic zero-copy model is very generic, and YaFF adopts it as well. Generated code builds a flat buffer, and generated API reads fields directly from it. Scalar values are stored inline in the object; variable-size data — strings, arrays, and nested messages — are stored as offsets.

The main complexity shifts to the layout. We need to decide exactly how to arrange scalars, nested objects, and metadata so that we simultaneously preserve Protobuf semantics, support schema evolution, and make reads as cheap as possible.

Accounting for the Server Environment

When developing YaFF we kept the target environment in mind. The format is designed not as a universal representation for arbitrary hardware, but as a runtime format for server hardware. YaFF therefore packs fields densely and adds no alignment padding inside the serialized representation.

Alignment simplifies portability and guarantees cheap access across a broad range of platforms, which makes it natural for more general-purpose formats. In our case, the platform set is limited to modern server x86_64 and ARM, which in typical scenarios handle unaligned reads efficiently.

Dense packing can make individual reads more expensive — for example, if a field crosses a cache-line boundary. But for large in-memory indexes, what matters is not just the speed of a single read but the total volume of data passing through caches and memory. By eliminating alignment padding we reduce that volume and, consequently, the pressure on the memory bus. In the end, we gain more than we lose on the occasional unaligned read.

This decision pairs well with YaFF buffer immutability. After construction the format is only read, so we pay nothing for the complexity of writing to unaligned fields and create no additional cache-coherence pressure.

Flat Layout: Almost a C++ Struct

The starting point for YaFF is the ordinary C++ struct. Field access in a struct is as efficient as it gets: the field’s location is known at compile time, and a read translates to a single memory access.

In the first variant of the serialized representation, called Flat Layout, we try to preserve this property of efficient access while adding minimal support for schema evolution.

This can be achieved with a surprisingly simple step: prepend the dense C++ struct with a 2-byte header that records the maximum field id present in the buffer, plus one.

The resulting metadata overhead over a raw struct is just 2 bytes — extremely compact. The read algorithm is maximally simple: read the header, compare the requested field’s id against the header, return the default if the id is larger, otherwise read the data.

For example, a header value of four means the buffer contains fields with id less than four. New code attempting to read field 5 immediately gets the default. Field 2 is read directly from the data at a pre-computed static offset.

In total, this approach performs two reads and one branch. Compared to FlatBuffers’ four reads and two branches, it has half the operations and no chain of dependent memory reads.

On the hierarchical benchmark this yields a 2–2.5× speedup.

With call-chain caching, Flat Layout comes close to raw C++ structs: 9.8 ns vs. 8.17 ns. The remaining gap is due to the unavoidable header check and the general Alias Analysis issue, which we’ll revisit shortly.

But this simplicity has a price. Flat Layout is fast precisely because every field’s offset is computed statically. For a field with id = N this means the layout must know the sizes of all N − 1 preceding fields and preserve their space in the physical representation. A field in the middle of the schema cannot simply be removed — that would shift all subsequent fields — and additions are only possible at the end. We therefore need to figure out how to reconcile this constraint with Protobuf semantics: field deletion, reserved, and presence.

Flat Layout: Adding Presence

One of the key aspects of Protobuf is its presence semantics. If a field is absent, its getter returns the default value defined in the schema. Beyond that, fields fall into two classes:

• implicit presence: the format does not separately record whether the field was initialized;
• explicit presence: initialization is recorded explicitly, and has_ methods appear in the API.

The exact semantics depend on the field type and the Protobuf version. For example, repeated fields always operate under the implicit model; nested messages use explicit presence; and the behavior of scalars depends on the version and the presence of the optional modifier. YaFF must support the same variety of combinations.

Returning the default value is straightforward. If the check id < header fails, the field lies beyond the recorded layout boundary, and the default can simply be compiled into the corresponding branch of the getter. If the check passes, the getter reads the value from the buffer.

For the second scenario, one could explicitly write the default value into the buffer. But there is a more convenient representation: store the field value as XOR with its default. This is computationally nearly free, and for fields with a zero default it reduces to a no-op. In return, we get a uniform physical representation for an uninitialized value: a block of zero bytes. Such gaps compress well even with fast codecs like LZ4, and are also easier to reuse when the layout evolves.

const auto encoded = value ^ default; // value ^ 0 == noop;
const auto value = encoded ^ default; // encoded ^ 0 == noop;

By default, Flat Layout does not store presence information: for fields with implicit presence, returning the correct value from the getter is sufficient. If a message has no explicit-presence fields, the layout stays the same.

For explicit presence, we need additional information about field presence, which can be encoded in a bitmask. This bitmask is needed only by has_ methods; getters don’t touch it and continue to read data directly, recovering the value via XOR with the default.

We place the bitmask to the left of the message header so that it potentially doesn’t push the payload data into a different cache line. This preserves Flat Layout’s efficiency: the presence of a bitmask has no effect whatsoever on the read algorithm.

The bitmask also doesn’t always need to be stored. For a has_ method, a value different from the default already proves presence: that field could not have appeared from an absent value. An extra bit is only needed for the single ambiguous state — when an explicit field was set but equals the default value. The presence bitmask therefore appears in the buffer only when at least one such field is encountered during serialization.

Flat Layout: Internal Dynamics

The optional presence bitmask requires an important property from the layout: dynamic selection of the read strategy. At runtime, we need to determine where to get presence information — from the bitmask or from the field value itself. Any such dispatch requires metadata and additional branches.

For has_ methods this is acceptable: they are called far less frequently than ordinary getters. But the primary value-read algorithm must not be touched. Flat Layout was built around fast reads, and presence must not add new operations to that path.

We therefore encode the bitmask flag directly in the two least-significant bits of the header. The combination 00 encodes the absence of a bitmask, and 01 encodes its presence (the second bit will be needed later).

The layout boundary itself is stored in the remaining 14 bits. This allows roughly 16,000 fields in a single message. Protobuf formally permits up to 65,535 fields, but for portable schemas the practical limit is lower: Java code generation, for example, hits JVM limitations at several thousand fields. So this trade-off barely constrains real messages while eliminating the need for separate metadata.

Crucially, ordinary getters get no additional work. The field identifier is known at code-generation time, so the constant used for comparison with the header is pre-encoded in the same format, and the low-order bits have no effect on comparisons.

The extra dispatch appears only in has_: if the bitmask flag is set, the method reads the presence bit; otherwise it derives presence from the value by comparing it against the default.

Flat Layout: Minimal Self-Describing Metadata

Protobuf has an important property: the serialized representation is partially self-describing. Without a schema we cannot reconstruct a complete object, but we can decompose the stream into fields: we can see a field identifier, wire type, and boundaries of variable-length values. This allows the parser to skip unknown fields and makes safe schema evolution possible. YaFF needs a similar mechanism.

As noted above, to statically compute the offset of a field with id = N in Flat Layout, we must know the sizes of all N − 1 preceding fields. If a field is deleted from the middle, we can no longer compute offsets for the fields that follow it. For writing new data this is no problem: after such a change the writer can choose a different layout, discussed below. But the reader must remain backward-compatible and be able to read old data that was written in Flat Layout.

The message must therefore store minimal metadata about the size of each field. During code generation, it is known which deleted identifiers precede a given field. This allows the code generator to pre-generate reads of only the metadata needed to calculate the real offset of that field.

The cost of this approach is a small number of extra reads. First, field deletions happen infrequently — typically one or two fields. Second, these reads are generated only for fields that follow the deleted ones. Third, this mechanism is only used when new code reads old data, i.e., during the migration window. The extra reads will therefore be few, won’t create noticeable overhead, and the core Flat Layout read path remains unchanged.

Since some metadata is unavoidable, it must be made compact. We exploit the fact that there are only three possible field sizes:

• 1 byte: bool scalars;
• 4 bytes: 32-bit scalars and offsets;
• 8 bytes: 64-bit scalars.

A field’s size can therefore be encoded in two bits. We store this data alongside the presence bitmask, and its presence is encoded in the second bit of the header. That is, the first two bits of the header encode the metadata volume per field in bits.

Flat Layout thus adaptively stores only the metadata a given message actually needs. If there is no explicit presence and no need to delete fields from the middle, metadata reduces to the base header. If has_ methods are needed, a presence bitmask is added. If old buffers must be readable after field deletion, field-size information is added.

As a result, Flat Layout supports the key properties of Protobuf’s wire format while keeping the most common read operation nearly as efficient as raw C++ structs.

Sparse Layout: Solving the Sparsity Problem

Flat Layout is optimized for one profile: dense data and maximally hot reads. Its efficiency rests on every field’s offset being statically known. But this same property defines its main limitation: the physical size of a message depends on the maximum field id, not on the number of populated fields.

If an object contains a field with id = N, Flat Layout reserves memory regions for all preceding ids. And here we encounter a two-level problem.

Sparse Data

First, regions must be reserved for unpopulated fields. When data is dense, most fields are populated, so the overhead is negligible. But when few fields are actually initialized, the message consists mostly of empty bytes that we are forced to store and transmit.

To see this effect in isolation from other factors, take a synthetic schema: 50 fields of 64 bits each, with random population at varying density.

Flat Layout saturates quickly: as soon as a large field id appears among the populated fields, the format is forced to store all preceding fields. At low and medium density it therefore loses noticeably to FlatBuffers on size; in this test the crossover point only appears near high fill rates, around 75%.

At the level of a single object the difference may look small: tens or hundreds of bytes. But in runtime indexes it is multiplied across millions of objects with dozens of nested structures, and becomes significant.

Sparse Schemas

Second, when a Protobuf schema contains a gap in field ids, we can no longer compute offsets for fields beyond the gap, so it is impossible to write data in Flat Layout.

For the hottest data, one can be disciplined about the schema and avoid id gaps, and when deleting a field leave size metadata for it in the schema. But in any system there are loads an order of magnitude smaller that don’t require such read efficiency.

YaFF therefore provides a second mode: Sparse Layout. Its trade-offs are similar to FlatBuffers’: we add a metadata table, which allows only actually present fields to be stored. But unlike switching directly to FlatBuffers, we remain in the same proto schema and the same protobuf-like API. This layout is slower than Flat Layout, but handles arbitrary Protobuf schemas, sparse objects, and scenarios where size and simplicity of evolution matter more.

Sparse Layout: Building on FlatBuffers’ vtable Ideas

For sparse data, a metadata table is a fairly natural trade-off. If we don’t want to store absent fields, we need to record somewhere which fields are present and where their values lie. Sparse Layout uses the same basic idea as FlatBuffers’ vtable, but adapts it to YaFF’s properties.

A message begins with the same 2-byte header as Flat Layout, preceded by an offset to the metadata table.

The shared header matters because it will later allow dynamic switching between Flat Layout and Sparse Layout. Storing the size in the message rather than in the metadata table provides locality: for a large table, this saves one cache miss. It also enables table deduplication not just in full, but by common prefix, since we no longer have a size field at the start of the table getting in the way.

We then exploit the fact that we don’t need to write fields to the buffer in arbitrary order. YaFF builds the buffer via generated code from an already-populated Protobuf message. The writer can therefore record fields in fixed id order. Fields that appear first will have small offsets, allowing their values to be represented more compactly.

The maximum physical field size in YaFF is 8 bytes. Offsets for the first 31 fields are therefore guaranteed to fit in one byte each: even in the worst case 31 × 8 = 248. From the 32nd field onwards that guarantee no longer holds, so 2-byte slots are used for all remaining fields.

The slot width in the table depends only on the field’s id and is known at code-generation time. The getter spends no runtime resources figuring out how many bytes to read: the required size is already compiled into the generated code.

This is effectively a static analogue of Protobuf’s varint idea: lower-numbered fields get a more compact metadata representation. But instead of a dynamic varint parser, we use a fixed layout and compile-time schema knowledge.

On the hierarchical benchmark, Sparse Layout predictably comes close to FlatBuffers.

Sparse Layout uses roughly the same class of algorithm as FlatBuffers: reading a field passes through the metadata table and an offset. So on small data volumes, when data fits in cache and the main resources are spent on instructions, branches, and dependent reads, the result is unsurprisingly close. The real value of Sparse Layout is not acceleration in this scenario, but a more compact metadata representation and compatibility with Protobuf semantics.

Sparse Layout: Reserved Fields

In Sparse Layout, some Protobuf semantics are easier to support than in Flat Layout, because the metadata table already describes a field’s presence and its position in the buffer.

A zero slot means the field is absent; a non-zero slot points to the data and simultaneously encodes presence, even if the field was explicitly set to the default value. The value’s size can be recovered from neighboring slots: the next non-zero offset defines the boundary of the current field.

The main advantage shows up for reserved fields and id gaps. In Flat Layout, a deleted field’s physical size must be known, otherwise all subsequent fields shift. In Sparse Layout, the deleted field’s type is not needed: a skipped id is simply an empty slot of fixed size in the table.

This is usually sufficient: large id gaps are rare, and an empty slot costs just one or two bytes. If a gap does become too large, the code generator can detect this and offer to set a compact internal layout-id via a proto option, keeping the original Protobuf identifier as the external contract.

Dynamic Layout: Another Nearly Free Dispatch

We now have two representations with different trade-offs. Flat Layout provides the cheapest possible reads for dense data and the hottest code paths. Sparse Layout serves as a flexible fallback for sparse objects and arbitrary Protobuf schemas.

What remains is to combine them in a way that the layout selection doesn’t slow down the most important scenario. Here we apply a principle similar to Huffman coding: the most frequent cases must have the shortest execution paths. For field reads, the hierarchy is:

1. Field is present in Flat Layout → the primary happy path.

2. Field is absent from Flat Layout → less frequent: data is dense, new fields are rare.

3. Sparse Layout → a deliberate fallback, where we pay with extra metadata for size and flexibility.

The selection algorithm must therefore be structured so that the first case incurs no additional operations compared to plain Flat Layout.

For this we again use the shared 2-byte header. In Flat Layout, the presence of extra metadata is encoded in the low-order bits specifically so as not to interfere with the first comparison. Now we want the opposite: a single first comparison that simultaneously checks two things — that we’re indeed looking at a Flat Layout and that the requested field falls within the recorded boundary. Let’s use the high-order bit for that.

This leaves even fewer useful bits for the layout boundary, reducing the maximum field count to roughly 8,000. In practice this is not a limitation, for the same reasons as before: portable Protobuf schemas typically hit code-generator constraints — for example, JVM limits — well before that.

The resulting algorithm is simple:

constexpr auto limit = ((id << 2) | 0x8000);
if (limit < Header_) {
   return ReadFieldFlat<T>(id);
}
if (Header_ & 0x8000) {
    return dflt;
}
return ReadSparseField<T>(id);

If the first comparison passes, we immediately read the field at its static offset with no additional checks over plain Flat Layout. If it fails but the header is still marked as Flat Layout, the field is absent and the getter returns the default. Only as a last resort do we fall through to Sparse Layout.

For Sparse Layout, this dispatch is nearly free: the field-access algorithm starts by reading the header, which we have already read. A few branches are added, but the number of memory accesses does not increase.

The final benchmark, taking into account Dynamic Layout looks like this:

These numbers confirm that Dynamic Layout adds almost no cost to the primary read paths.

The Flat path remains as fast as before, and Sparse remains comparable to FlatBuffers.

A Few Notes on the Generated API

The resulting layout contains all the information needed for Protobuf semantics. Generating a protobuf-like API for zero-copy access on top of it — along with serialization methods — is a fairly straightforward task. The most interesting part is the safe and efficient support for accessor chains discussed earlier.

In Protobuf, such code is safe: if an intermediate message is absent, the getter returns a default instance, not a null pointer. YaFF needs an equivalent object. It is a 2-byte header with the Flat bit set and a zero layout boundary. In such an object, the presence check for any field fails immediately, so the getter returns the default at once. This means an absent nested message can be read without materializing an object and without a cascade of null-dereference checks.

Another task is making accessor chains legible to the optimizer. If code accesses the same intermediate object multiple times — say req.context().user() — it would be ideal to evaluate that accessor once and reuse the result when reading multiple fields.

For this, the generated buffer-read functions are annotated with gnu::pure. The documentation describes this attribute as:

“Many functions have no effects except the return value and their return value depends only on the parameters and/or global variables. Such a function can be subject to common subexpression elimination and loop optimization.”

For YaFF this contract fits well: the buffer is immutable, we only read data and do not change state. The attribute therefore gives the optimizer a useful local hint: a repeated call with the same arguments has no side effects and reads from the same memory.

That said, gnu::pure does not guarantee that a chain will always be collapsed. It is just one signal to the optimizer; the actual outcome depends on the surrounding code — Alias Analysis, visible stores between calls, and whether the relevant memory could be proven unchanged. But in many cases it is enough to make the compiler reuse intermediate computations instead of re-traversing the same chain for each field.

As a result, with all these refinements, we get behavior close to raw C++ structs for a significant number of use cases, and a solid fallback for the rest.

YaFF — A Dynamic Zero-Copy Wire Format for Protobuf

That concludes the story about byte layout. We wanted to read Protobuf data without deserialization while retaining the proto schema as a contract, the familiar API, default values, presence, reserved, and safe schema evolution.

To achieve this, YaFF uses not a single universal layout but an adaptive data representation model. Flat Layout handles the hottest and densest data: minimal metadata and a read algorithm close to C++ struct access. Sparse Layout handles sparse objects and flexible schema evolution: a metadata table, storing only present fields. Dynamic Layout ties them together while preserving Flat Layout’s efficiency.

From the outside, this doesn’t change the development model: the developer still sees the proto schema and a protobuf-like API. Internally, YaFF picks the physical representation to match the data: fast and dense for the hottest services, flexible and compact for everything else.

For the developer, it’s still Protobuf. For the runtime, it’s a zero-copy representation.

How to Try YaFF

YaFF integrates into a C++ project via CMake or Conan: the code generator is configured alongside ordinary Protobuf and works directly with your existing proto schemas. Our Quick Start guide walks you from schema definition to zero-copy reads in minutes.

YaFF’s code is available on GitHub under the Apache 2.0 license, along with detailed documentation, examples, and benchmarks.

The project is early in its life and actively evolving, so there will likely be bugs here and there — we will fix them. If you run into issues or realize the format is missing an important feature, please open an issue. And if you’d like to propose an improvement yourself, pull requests are always welcome.

Conclusion: The Data Interface Doesn’t Have to Dictate the Physical Representation

The main takeaway from YaFF goes beyond any specific format. For data, just as for code, the interface need not dictate the implementation.

We are accustomed to thinking of the Protobuf ecosystem as a monolith: proto schema, wire format, and parser. YaFF separates these layers. Schemas remain the contract between teams and services; the protobuf-like API remains the familiar developer interface. The serialized representation becomes a separate backend that can be chosen to match the specific workload.

We covered two such backends in detail — Flat Layout and Sparse Layout — for zero-copy runtime access. But we’re already working on the next step: a columnar layout inside YaFF. It fits into the same model as yet another backend: large repeated fields with arbitrary nesting can be stored compactly in a form better suited to analytics and ML pipelines.

The ability to preserve a shared proto contract and plug in specialized backends precisely where they pay for themselves is especially important for large systems, where you cannot stop the world and rewrite tens of thousands of lines of code for a new format.

Protobuf remains the language in which a system agrees on its data. YaFF makes the physical representation of that data a replaceable, optimizable layer. We’d love for you to try YaFF in practice and share your impressions!