DataBytes: Data Encodings

Photo by Markus Winkler on Unsplash

The difficulty of getting people to agree on anything outweighs most other concerns.

This reality becomes especially apparent when systems need to exchange data across computers and services — and even more so when those exchanges involve gigabytes or terabytes of information. In distributed systems, consensus is often harder than computation itself.

There are (at least) two data representations that programs work with:

1. In-memory data structures like arrays, lists, trees, hash tables, structs, etc. which are optimized for fast access and manipulation by the CPU.
2. Files saved on the disk and sent over the network like JSON strings which are self-contained sequences of bytes in the binary format.

There is often a need to move between these representations. The process of converting in-memory data objects to binary representation is called serialization or encoding and the reverse process is called deserialization or decoding.

There are several serialization methods available to translate in-memory data into byte sequences. We can classify them into the following two classes:

1. Language-specific like pickle for Python and Marshal for Ruby which provide easy conversion with minimal code overhead. These formats are tied to a specific programming language and need to create arbitrary classes on deserialization introducing a security risk. They can have bloated file sizes as they are intended only for quick and easy encoding and decoding of data.
2. Language agnostic standard formats like JSON, XML and CSV provide higher compatibility and smaller file sizes. These are widely adopted and mostly have human readable textual formats. However, these formats have their own limitations as highlighted below.

Cons of using JSON

• Does not distinguish between integers and floats making it difficult to store integers greater than 2⁵³ as they cannot be represented accurately in the double precision format. In fact Twitter had to include the tweet ids twice, once as an integer and once as a string, in order to prevent issues with parsing.
• Does not support binary data. Instead they have to be encoded as base-64 strings which increased the file size by 33%.

Cons of using XML

• Does not distinguish between numbers and strings consisting only of digits.
• Verbose as compared to JSON.
• Does not support binary data.

Cons of using CSV

• Does not distinguish between numbers and strings consisting only of digits.
• It is schemaless and the application code needs to assign meaning to each column.
• New column and row additions need to be handled manually.
• Not all parsers implement the CSV parsing rules consistently.

Though JSON, XML and CSV remain popular, the crux of using the right data format boils down to the following:

As long as people agree on what the format is, it does not matter how pretty or efficient the format is.

References:

1. M. Kleppmann, Designing Data-Intensive Applications: The Big Ideas Behind Reliable, Scalable, and Maintainable Systems. O’Reilly Media, 2017.