From 128 Codes to 1.1 Million: A Journey Through Text Encoding

Anyone who has worked in software engineering has heard words like encoding, ASCII, Unicode, and UTF-8. But have you ever wondered what do these terms really mean? How have we arrived at one standard accepted by the whole internet ?

If yes, then just read along while we get into a bit of history of encoding and determine how UTF-8 came to be one of the most popular format on the internet.

And like me if you’re a fan of Japanese words that capture deep concepts — like Ikigai ,Wabi-Sabi etc. — you’ll learn a new one here.

Introducing ASCII

We speak different languages and write characters, emojis etc. but Computers, only deal with binary 0s & 1s. So how do we make computers store and process text ?

The earliest popular attempt to bridge this gap was ASCII (American Standard Code for Information Interchange), formalized in the 1960s. Its approach was straightforward:

• You have 26 letters in the English alphabet, double that for uppercase and lowercase so 52 codes.
• Add 10 digits leading to 62 codes.
• Add punctuation , special symbols, and the total quickly crosses 64 (2⁶).

so 7 bits (giving 128 possible codes, leaving room for some possible extensions).

But ASCII had a major weakness: it was very America-focused. For example, it included $ but not £. Covering other languages, symbols, or the emojis we use today was far beyond.

So what came next ?

Patch Work : ISO-8859 Family

Different regions extended ASCII in different ways e.g. Europe had ISO-8859–1 (Latin-1), which added support for accented characters like é, ñ, and ü. Other regions rolled out their own extensions.

The problem? These encodings clashed. A byte value like 0XE9 might mean “é” in Latin-1, but something completely different in another encoding.

This caused several issues when systems tried to talk to each other. Ever opened a file and seen gibberish instead of readable text ?

That mess has a name in Japanese: Mojibake (文字化け), literally “character transformation.” is the garbled or gibberish text that is the result of text being decoded using an unintended character encoding. (see wikipedia)

Soon major computer companies realised that extending ASCII might not be the solution and they began developing an alternative to ASCII called **Unicode.**

So What’s Unicode and how it solved the problem ?

Unicode is a character encoding scheme that aims to consistently represent every character from all writing scripts, symbols emojis and more.

It assigns a code point to every character which is nothing but a number which is usually written in hexadecimal (as it’s a concise way of representing large numbers). Below are some examples of unicode mappings of characters.

• Character = A → Unicode Code Point = U+0041 → Decimal = 65
• Emoji = 😀 → Unicode Code Point = U+1F600 → Decimal = 128512

Now Unicode created a giant catalogue that provided a numerical conversion of everything but it only defines what characters exist and their unique numbers. It doesn’t say anything about how code points are stored in memory or transmitted over network.

That’s where the UTFs(Unicode Tranformation Formats) come in.

UTF: Unicode Transformation Format

Unicode in it’s latest version uses 21 bit address space and that leaves it with approx 1.1 million code points (U+0000 to U+10FFFF).

Originally, Unicode was designed as a 16-bit code, with the vision of “one character = one 16-bit word.” This gave it a total space of 2¹⁶ = 65,536 code points, covering the range U+0000 to U+FFFF.

That made sense because 16 bits could cover 65,536 code points, which was thought to be enough for “all the world’s characters.”

But soon we started using much richer text including emojis plus there were a lot of historical scripts that were needed to be included hence they decided to extend beyond 65,536 code points.

Now one solution is stop following the earlier assumption of 1 char=16 bit but by then many programming languages, operating systems etc. had already locked in that assumption so they took an interesting approach of using surrogate pairs that helped maintain the 16 bit design.

Surrogate pairs use two 16-bit words (from special reserved ranges: U+D800–U+DBFF for high surrogates, and U+DC00–U+DFFF for low surrogates). This allows representing code points up to U+10FFFF (1,114,112 possible values) using 21 bits in total.

Different encoding that were designed to support storage and transmission:

• UTF-32: Represents every Unicode code point using 4 byes or 32 bits. Takes a lot of space & 11 bits are wasted for every character as Unicode uses max. 21 bits.
• UTF-16: represents Unicode characters until U+FFFF as 2 bytes and characters above U+FFFF as 4 bytes. Again this format takes 2 bytes for ASCII characters that can be represented using one byte.
• UTF-8 : Most important unicode transformation format and used by 97% of all web pages. It is backward compatible with ASCII i.e any file consisting solely of 7 bit ASCII codes and stored as bytes is automatically a UTF-8 file.

UTF-8 is a variable length transformation format that stores ascii characters as 1 bytes and all others as 2,3 or 4 bytes. Below is how it works:

[embed]UTF-8 : Unicode ranges

UTF-32 and UTF-16 are still used by some systems: UTF-32 in situations where fixed-width encoding is needed, and UTF-16 in certain platforms like Windows and Java. But on the open web, they are rare because of their storage overheads and complexity. UTF-8, in contrast, provides a very pragmatic approach towards storage and transmission of Unicode code points, which is why it has become the most popular choice.

Endianness

One other issue that can come up with exchange of data between systems is based on Endianness : the order in which bytes are stored.

Big-endian : Most significant byte first (like how we write numbers left to right).

Little-endian : Least significant byte first.

Imagine two machines exchanging the 16-bit value 20h ACh (unicode for euro), it will be:

• 20h ACh (Euro) on a big-endian system.
• ACh 20h (a Korean Character) on a little-endian system.

If both sides assume different orders it will mean very different things.

So, to get around this problem unicode introduced a Byte Order Mark (BOM) supposed to be placed at the begining of the file. If it’s FEh FFh it’s big endian , else if it’s FFh FEh it’s little endian.

UTF-8 technically doesn’t need a BOM because byte order doesn’t apply, but some files may include it (EF BB BF).

Summary

From ASCII’s 128-code beginnings to Unicode’s 1.1 million code points, the journey of text encoding is full of trial, error, and clever engineering. We saw how different regions extended ASCII, ran into mojibake, and why Unicode had to separate the idea of characters from their storage (UTF-8, UTF-16, UTF-32). We learned lessons about byte order, surrogate pairs, and why pragmatic choices like UTF-8’s variable-length encoding made it the web’s favorite.

Next time you type 😀 into a chat and it just works, pause for a second. Behind that smiley is decades of messy history, clever design, and engineers agreeing (after much chaos) on one universal way to talk.