Slash Your LLM API Costs: An Introduction to TOON, the JSON Alternative

If you’re building applications with Large Language Models (LLMs), you know that every token counts. The cost of API calls is directly tied to the number of input and output tokens, and as your application scales, these costs can quickly add up. But what if there was a way to send the same data to your LLM while using significantly fewer tokens?

Enter TOON (Token Oriented Object Notation), a new data format that’s rapidly gaining popularity. This trending GitHub project presents itself as a “JSON for LLM prompts at half the tokens,” and it delivers on that promise. In this post, we’ll explore what TOON is, why it’s so effective, and how you can use it to reduce your LLM costs and improve performance.

The Problem with JSON in LLM Prompts

For years, JSON has been the de facto standard for structuring and transmitting data. It’s human-readable, widely supported, and easy to work with. However, when it comes to feeding data into an LLM prompt, its structure can be surprisingly inefficient.

LLMs don’t see characters; they see tokens. Every curly brace {}, square bracket [], comma, quote, and even whitespace character in a JSON object is counted as one or more tokens. Consider a simple array of objects:

[
 { “id”: 1, “name”: “Alice”, “role”: “admin” },
 { “id”: 2, “name”: “Bob”, “role”: “user” }
]

In this example, you are sending a lot of repetitive, structural characters. The keys (”id”, ”name”, ”role”) are repeated for every single object. If you have a thousand objects, you’re sending the same keys a thousand times, along with 2,000 curly braces, plus countless quotes and commas. In a real-world scenario with a million objects, this translates to millions of extra, redundant tokens that you’re paying for.

How TOON Solves the Token Problem

TOON was designed specifically to eliminate this syntactic overhead. It transforms structured data into a highly compact, token-efficient format that is still easily understood by an LLM.

Instead of repeating keys for every entry, TOON defines the structure once and then lists the data in a clean, tabular format. The JSON example from above would look something like this in TOON:

users(2):
| id | name | role |
| 1 | Alice | admin |
| 2 | Bob | user |

(Note: This is a conceptual representation of the TOON format)

Notice the difference? The keys are defined only once. The curly braces, commas, and repetitive quotes are gone. This minimalist, indentation-based structure dramatically reduces the character count, which in turn leads to a significant reduction in tokens.

The Proof: A Real-World Cost Comparison

Talk is cheap, but API calls aren’t. Let’s look at a practical test comparing the token usage of JSON versus TOON when sent to the OpenAI API.

In a test where the same dataset was provided to an LLM, first in JSON and then in TOON, the results were clear:

With JSON:

• Prompt Tokens: 149
• Estimated Cost: $0.001765 With TOON:
• Prompt Tokens: 135
• Estimated Cost: $0.001200

This simple test resulted in 9.4% fewer input tokens with TOON, leading to a direct cost saving. The TOON repository itself claims savings can be anywhere from 30% to 60%, which is a huge achievement for any developer working with LLMs at scale.

Getting Started and an Important Caveat

Implementing TOON in your project is incredibly straightforward. If you’re using JavaScript or Node.js, you can get started with a simple command:

npm install toon

From there, you just need to import the library and use the encode function to convert your JSON data into the TOON format before including it in your LLM prompt.

import toon from ‘toon’;

const jsonData = [ { “id”: 1, “name”: “Alice” } ];
const toonString = toon.encode(jsonData);

// Now, use toonString in your LLM prompt

A Word of Caution: TOON is most effective for flat or tabular data. The video’s creator points out that for deeply nested or non-uniform data structures, traditional JSON might actually be more token-efficient. In such cases, the recommended approach is to first “flatten” your nested object into a flat structure before converting it to TOON.

Conclusion: A Smarter Way to Prompt

TOON offers a compelling solution to the ever-present problem of high LLM API costs. By rethinking how we structure data for machine consumption, it provides a simple yet powerful way to optimize our prompts, save money, and potentially even speed up response times.

If you’re frequently sending structured data like database records or user information to an LLM, TOON is a tool you should seriously consider. It’s a small change in your code that could lead to significant savings in your budget.