Compare JSON Files with JSONL Files in Python

Learn a memory-efficient way to save large datasets

In this post, we will compare JSON (.json) and JSON Lines (.jsonl) file formats with simple examples. Both are used to store structured data primarily for data exchange. We will learn which one is more memory-efficient to save large datasets.

Image by kreatikar in Pixabay

[embed]https://youtube.com/@kwonglynn

JSON files

Most developers should already be familiar with JSON files, which are used to store a single JSON object or array. A JSON object is a dictionary in Python, and an array is a list of dictionaries.

[
    {"id": 1, "value": "A"},
    {"id": 2, "value": "B"}
]

As shown in this example, the data stored in the example.json file is a single array that contains two JSON objects.

We can use the load() function from the json module to read JSON files. Note that all the data is read at once and is thus inefficient for large data set.

import json

# Read JSON file.
with open("example.json", "r") as file:
    json_data = json.load(file)

print(json_data)
# [{'id': 1, 'value': 'A'}, {'id': 2, 'value': 'B'}]

When writing to JSON files, we can use the dump() function from the json module to dump a Python dictionary or list to a JSON file.

import json

data = [
    {"id": 1, "value": "A"},
    {"id": 2, "value": "B"},
]

with open("out.json", "w") as file:
    json.dump(data, file)

JSONL files

On the other hand, JSONL (which is short for JSON Lines) files contain multiple JSON objects separated by newlines, with each line representing a valid JSON object. Note that no matter how big the JSON object is, it must be put in a single line.

{"id": 1, "value": "A"}
{"id": 2, "value": "B"}

JSONL files can be more memory-efficient for large datasets because we can read each line one at a time.

Unlike JSON files, we need to use the loads() function from the json module to read each line of a JSONL file. Since each line is a valid JSON, it can be read independently.

import json

# Read JSONL file line by line.
with open("example.jsonl", "r") as file:
    for line in file:
        data = json.loads(line)
        print(data)

# {'id': 1, 'value': 'A'}
# {'id': 2, 'value': 'B'}

As a consequence, JSONL files are more suitable to store large data set as we don’t need to read everything into memory altogether as with JSON files.

When writing a JSONL file, we need to call the dumps() function of the json module to convert each data entry to a JSON string and then write it to the target JSONL file.

import json

data = [
    {"id": 1, "value": "A"},
    {"id": 2, "value": "B"},
]

with open("out.jsonl", "w") as file:
    for entry in data:
        file.write(json.dumps(entry) + "\n")

Note that we need to add the new line separator for each data entry explicitly.

It should be noted that JSON files and JSONL files cannot be used interchangeably in most cases. However, some systems may not be very strict about the file extensions and may store JSONL files with the .json extension. If a service requires a JSONL file and a JSON file is used, you will get an error, and vice versa.

Related posts

• Common reasons for JSONDecodeError in Python
• Compare str(obj) and json.dumps(obj) in Python

In Plain English 🚀

Thank you for being a part of the **In Plain English** community! Before you go:

• Be sure to clap and follow the writer ️👏️️
• Follow us: **X | [LinkedIn](https://www.linkedin.com/company/inplainenglish/) | [YouTube](https://www.youtube.com/channel/UCtipWUghju290NWcn8jhyAw) | [Discord](https://discord.gg/in-plain-english-709094664682340443) | [Newsletter](https://newsletter.plainenglish.io/) | [Podcast](https://open.spotify.com/show/7qxylRWKhvZwMz2WuEoua0)**
• **Create a free AI-powered blog on Differ.**
• More content at **PlainEnglish.io**