A Programmer’s Guide to Unicode: Understanding Text Encoding and Its Importance

Unicode is a cornerstone of modern computing, enabling us to represent text from virtually every language and symbol system in the world. In this guide, we’ll explore what Unicode is, how it works, and why it’s essential for programming. Along the way, we’ll clarify key concepts like graphemes, code points, and encoding schemes, and provide practical examples.

Introduction: Why Unicode Matters

Text is everywhere in programming — whether it’s user input, file names, or web content. But representing text in computers isn’t as simple as it seems. Early systems like ASCII worked well for English, but they couldn’t handle the diversity of characters used in other languages or symbols like emojis. Unicode was created to solve this problem by providing a universal standard for text representation.

To understand Unicode, we need to start with the basics of how data is stored in computers.

All Data Is Stored As Bits

At its core, all data in a computer — whether it’s numbers, text, or images — is stored as bits (zeros and ones). For example:

• The number 26 is stored as its binary equivalent: 11010.
• But what about letters like D or symbols like 😊? How do we store them?

The solution lies in mapping characters to numeric values, which brings us to the concept of character encoding.

How Characters Were First Stored: ASCII

The earliest and simplest encoding system was ASCII (American Standard Code for Information Interchange). ASCII maps 128 characters (English letters, digits, and symbols) to numeric values between 0 and 127. For example:

• A → 65 → 01000001 (binary)
• B → 66 → 01000010 (binary)

Each character in ASCII is stored as 1 byte (8 bits). This simplicity made ASCII easy to use, but it had a major limitation: it could only represent 128 characters, which is insufficient for non-English languages.

Example: ASCII Encoding

Here’s how the word “HELLO” is represented in ASCII:

Key Property of ASCII

• Length in bytes = Number of characters: Since each character is stored as 1 byte, the len() function in many programming languages (e.g., C, Go) returns the number of bytes, which equals the number of characters for ASCII strings.

The Problem With ASCII: What About Other Languages?

Languages like Chinese, Arabic, and Hindi use thousands of characters, far exceeding ASCII’s 128-character limit. To address this, the Unicode Standard was created. Unicode encompasses over 100,000 characters from hundreds of languages, as well as symbols, emojis, and modifiers.

However, Unicode is more complex than ASCII. To understand it, we need to clarify some key concepts.

What Is a Grapheme?

A grapheme is the smallest unit of a writing system that humans perceive as a single character. For example:

• D (Latin letter)
• 你 (Chinese character)
• 😊 (emoji)

Think of graphemes as the symbols you’d place on a Scrabble tile.

What Is a Code Point?

In Unicode, each grapheme is assigned one or more code points, which are numeric values. For example:

• D → Code Point: U+0044
• 你 → Code Point: U+4F60
• 😊 → Code Point: U+1F60A

Some graphemes, like accented letters (é), can be represented by multiple code points (e.g., e + combining accent).

Encoding Code Points: From Numbers to Bytes

Code points are not stored directly in memory. Instead, they are encoded into bytes using encoding schemes. Unicode supports multiple encoding strategies, each with its own trade-offs. Let’s explore two popular ones: UTF-32 and UTF-8.

UTF-32: Simple but Wasteful

UTF-32 encodes each code point as 4 bytes (32 bits), regardless of its value. For example:

• D → 00000000 00000000 00000000 01000100
• 😊 → 00000000 00000001 11110110 00001010

While UTF-32 makes indexing easy (each code point is at a fixed offset), it’s highly inefficient for text with many small code points (e.g., English text), as it uses 4 bytes for every character.

UTF-8: Efficient and Backward-Compatible

UTF-8 is a variable-length encoding scheme that uses 1 to 4 bytes per code point:

• Small code points (e.g., ASCII characters) use 1 byte.
• Larger code points (e.g., emojis) use 2 to 4 bytes.

For example:

• D → 01000100 (1 byte)
• 😊 → 11110000 10011111 10011000 10101010 (4 bytes)

UTF-8 is space-efficient and backward-compatible with ASCII, making it the most widely used encoding today. However, its variable-length nature makes indexing more complex.

Unicode in Practice: Examples in Go

Let’s look at some practical examples in Go to understand how Unicode works. Before that let’s remember, in go:

• Strings are stored as a sequence of bytes, and the string type is an alias for a slice of bytes ([]byte)
• Go strings are UTF-8 encoded by default, using 1 to 4 bytes per character.
• A rune is an alias for int32 and represents a Unicode code point.

Example 1: Bytes vs. Runes (UTF-8)

In Go, strings are sequences of bytes. To work with Unicode, you need to use runes, which represent code points.

package main

import (
    "fmt"
)

// UTF-8
func main() {
    str := "A😊你"

    // Length in bytes
    fmt.Println("Bytes:", len(str)) // Output: 8(since "A" is 1 byte, "😊" is 4 bytes, and "你" is 3 bytes)

    // Length in runes (code points)
    runes := []rune(str)
    fmt.Println("Runes:", len(runes)) // Output: 3 (each character is a single rune/code point)

    // Print index, rune, and code point
    // Index: 0, Rune: A, Code Point: U+0041
    // Index: 1, Rune: 😊, Code Point: U+1F60A
    // Index: 5, Rune: 你, Code Point: U+4F60
    for i, r := range str {
        fmt.Printf("Index: %d, Rune: %c, Code Point: U+%04X\n", i, r, r)
    }
}

[0] = 0x41 (A) [1] = 0xF0 (😊) [2] = 0x9F (😊) [3] = 0x98 (😊) [4] = 0x8A (😊) [5] = 0xE4 (你) [6] = 0xBD (你) [7] = 0xA0 (你)

Example 2: Safe String Slicing

You can iterate over a string’s runes using a for loop.

package main

import (
    "fmt"
)

func main() {
    str := "A😊你"

    // Unsafe slicing (by bytes)
    fmt.Println("Unsafe:", str[:2]) // Output: A (may corrupt multi-byte characters)

    // Safe slicing (by runes)
    runes := []rune(str)
    fmt.Println("Safe:", string(runes[:2])) // Output: A😊
}

Unicode Rules of Thumb

• Use Unicode-aware string functions: Always use functions that understand Unicode to avoid corrupting multi-byte characters.
• Understand the difference between bytes, code points, and graphemes: Bytes are storage units, code points are Unicode values, and graphemes are human-perceived characters.
• Be cautious with slicing: Slicing strings without considering Unicode can lead to corrupted or unreadable text.

Conclusion on Unicode using UTF-8 in Go: In Go, strings are stored as sequences of bytes and are UTF-8 encoded by default. This encoding uses 1 to 4 bytes per character, allowing efficient representation of a wide range of Unicode characters. However, slicing strings by bytes can lead to issues with multi-byte characters, potentially corrupting them.

To handle Unicode characters safely, Go provides runes, which represent Unicode code points. Converting a string to a slice of runes ([]rune) allows for safe slicing and manipulation of strings containing multi-byte characters, ensuring that each character is treated correctly.

Conclusion

Unicode is essential for modern programming, enabling applications to handle text in any language. By understanding concepts like graphemes, code points, and encoding schemes, you can write Unicode-aware programs and avoid common pitfalls. Experiment with your favorite programming language to see how it handles Unicode and deepen your understanding of this critical topic.