NDJSON Explained: The JSON Format Built for Big Data and Streaming

If you’ve worked with APIs, log processing, Apache Spark, Kafka, Elasticsearch, or large-scale ETL pipelines, you’ve probably encountered NDJSON. While it looks deceptively simple, NDJSON solves several practical problems that traditional JSON struggles with at scale.

In this article, we’ll explore what NDJSON is, why it exists, how it differs from traditional JSON, and where it fits into modern data architectures.

What is NDJSON?

NDJSON (Newline Delimited JSON) is a format where each line contains a valid JSON object.

Instead of wrapping records inside a JSON array, every record is stored independently on its own line.

Traditional JSON

[
  {
    "id": 1,
    "name": "John"
  },
  {
    "id": 2,
    "name": "Jane"
  },
  {
    "id": 3,
    "name": "Bob"
  }
]

NDJSON

{"id":1,"name":"John"}
{"id":2,"name":"Jane"}
{"id":3,"name":"Bob"}

The concept is simple:

One JSON document per line.

This seemingly small change makes a huge difference when processing large volumes of data.

Why Was NDJSON Created?

Traditional JSON is excellent for configuration files and small API responses. However, it becomes inefficient when dealing with:

• Massive datasets
• Continuous data streams
• Log aggregation systems
• Distributed computing frameworks
• Real-time event processing

Consider a 50 GB JSON file stored as a giant array.

Before processing the records, many parsers must first understand the entire structure, including the opening bracket, commas between objects, and closing bracket.

With NDJSON, each record can be processed independently as soon as it is read.

This enables true streaming and incremental processing.

Visual Comparison

Traditional JSON

[
  {record1},
  {record2},
  {record3},
  ...
]

NDJSON

{record1}
{record2}
{record3}
{record4}
...

In traditional JSON, records are dependent on the surrounding array structure.

In NDJSON, every record is self-contained.

Key Advantages of NDJSON

1. Streaming-Friendly

One of the biggest advantages of NDJSON is its compatibility with streaming systems.

Imagine a pipeline like:

Application → Kafka → Spark → Data Lake

As events arrive, they can immediately be written as NDJSON records.

There is no need to wait for the entire dataset before processing begins.

This makes NDJSON ideal for:

• Event-driven architectures
• Real-time analytics
• Log aggregation
• Message processing systems

2. Memory Efficient

Traditional JSON arrays often require large portions of the file to be loaded into memory.

For example:

[
  1000000 records
]

With NDJSON, systems process one record at a time:

Record 1
Record 2
Record 3
...

This significantly reduces memory consumption and improves scalability.

3. Easy to Append

Appending new data to a JSON array can be cumbersome because commas and closing brackets must be managed carefully.

With NDJSON, adding a record is trivial:

{"id":4,"name":"Alice"}

Simply append another line.

No restructuring required.

4. Better Parallel Processing

Distributed computing frameworks thrive on NDJSON.

Because each line is independent, files can be split into chunks and processed simultaneously across multiple workers.

This makes NDJSON particularly effective for:

• Apache Spark
• Apache Flink
• Apache Beam
• Hadoop ecosystems

NDJSON in Real-World Logging

Application logs are one of the most common use cases for NDJSON.

Example:

{"timestamp":"2025-06-20T10:00:00Z","level":"INFO","message":"User Login"}
{"timestamp":"2025-06-20T10:00:02Z","level":"INFO","message":"Product Viewed"}
{"timestamp":"2025-06-20T10:00:04Z","level":"ERROR","message":"Payment Failed"}

Benefits include:

• Easy ingestion into analytics systems
• Simple searching and filtering
• Compatibility with streaming platforms
• Efficient storage and transport

Many modern observability platforms rely on newline-delimited formats for precisely these reasons.

Reading NDJSON in Python

Python provides straightforward support for NDJSON.

import json

with open("employees.ndjson") as file:
    for line in file:
        record = json.loads(line)
        print(record["name"])

Output:

John
Jane
Bob

Notice that the entire file never needs to be loaded into memory.

Reading NDJSON in Apache Spark

One reason NDJSON is popular among data engineers is that Spark can read it directly.

Scala Example

val df = spark.read
  .json("employees.ndjson")

df.show()

PySpark Example

df = spark.read.json("employees.ndjson")
df.show()

Spark automatically treats each line as an independent JSON document.

This makes NDJSON a natural fit for distributed data processing.

NDJSON and Elasticsearch

Elasticsearch uses newline-delimited structures for high-performance bulk ingestion.

Example:

{"index":{"_index":"users"}}
{"id":1,"name":"John"}

{"index":{"_index":"users"}}
{"id":2,"name":"Jane"}

Because records are processed sequentially, Elasticsearch can index data much more efficiently than processing a massive JSON array.

Common Use Cases

Data Engineering

• ETL pipelines
• Data lakes
• Batch processing
• Distributed analytics

Streaming Systems

• Kafka events
• Event sourcing
• Message queues
• Real-time processing

Logging and Observability

• Application logs
• Server logs
• Audit logs
• Monitoring systems

API Exports

Many modern APIs provide NDJSON export options because consumers can begin processing data immediately rather than waiting for a complete JSON payload.

When Should You Use NDJSON?

Use NDJSON when:

✅ Processing large datasets

✅ Building ETL pipelines

✅ Working with Spark or Kafka

✅ Streaming data between services

✅ Storing application logs

✅ Appending records frequently

Traditional JSON remains the better choice when:

✅ Returning small API responses

✅ Building configuration files

✅ Human readability is the primary concern

Final Thoughts

NDJSON is one of those simple ideas that delivers an outsized impact in modern data engineering.

By treating every line as an independent JSON document, NDJSON enables:

• Efficient streaming
• Lower memory consumption
• Easy appends
• Better parallel processing
• Scalable analytics pipelines

As organizations continue to process larger datasets and adopt real-time architectures, NDJSON has become a preferred format for moving, storing, and processing structured data.

If you’re working with Spark, Kafka, Elasticsearch, cloud data platforms, or large-scale ETL systems, mastering NDJSON is a small investment that pays dividends across nearly every data engineering project.