Character Encoding in Java Strings

Image Source

Java uses UTF-16 encoding to store characters in its String class, which makes it compatible with a large range of Unicode characters. But this also comes with challenges when handling special characters like emojis and symbols from different languages. This beginner-friendly article looks at how Java processes characters in memory, how multi-byte characters such as emojis are stored, and how encoding mismatches can cause display issues.

How Java Represents Characters in UTF-16

Java’s String class uses UTF-16 encoding, meaning each character is stored as one or two 16-bit code units. This system provides a way to support a vast range of characters from different writing systems while keeping memory usage relatively efficient compared to older fixed-width encodings. However, the way Java handles characters at a low level affects how they are stored, retrieved, and processed in applications.

Storing Basic Characters

Most commonly used characters, including standard English letters, digits, and punctuation, fit within a single 16-bit unit. These characters have Unicode code points ranging from U+0000 to U+FFFF, which means they map directly to a single 16-bit value in memory.

For example, the letter A (U+0041) is stored in UTF-16 as:

Hex: 0041  
Binary: 00000000 01000001

The same applies to other standard ASCII characters, which all fit within a single 2-byte space. This makes reading, writing, and processing these characters straightforward, as each one occupies exactly one slot in memory.

Storing Characters Beyond the Basic Range

While UTF-16 is often thought of as a fixed-width encoding, it actually uses a variable-length format. Some characters, particularly those from extended scripts like ancient languages or modern emojis, require more than a single 16-bit unit.

Characters with Unicode code points beyond U+FFFF fall into the supplementary range, which requires two 16-bit units to be stored. This is done using surrogate pairs, a mechanism that allows UTF-16 to encode characters beyond what fits in a single 16-bit space.

For example, the 😂 emoji has a Unicode code point of U+1F602. Since this is larger than U+FFFF, it is broken into a pair of values:

D83D DE02 (Hex)

In binary, this looks like:

D83D (11011000 00111101) -> High Surrogate  
DE02 (11011110 00000010) -> Low Surrogate

Instead of being stored as a single value, Java splits it into two separate 16-bit values that must be processed together to represent the full character.

How Java Processes Surrogate Pairs

While a single char in Java is always 16 bits, not all characters fit within one char. This means that a String containing characters beyond U+FFFF will have a length() greater than the number of actual characters visible.

Example in Java:

public class UnicodeExample {
    public static void main(String[] args) {
        String emoji = "😂"; // Unicode U+1F602
        System.out.println("Length: " + emoji.length()); // Outputs 2
    }
}

Even though the string only contains one character, Java reports a length of 2 because the emoji is stored as two separate 16-bit code units.

To correctly process such characters, Java provides methods like codePointAt() and codePointCount() to work with full Unicode code points instead of just individual char values.

Example:

public class UnicodeCodePoints {
    public static void main(String[] args) {
        String text = "A😂B"; // Contains a mix of single and multi-unit characters
        System.out.println("Total length: " + text.length()); // Outputs 4
        System.out.println("Code Point Count: " + text.codePointCount(0, text.length())); // Outputs 3
    }
}

Here, the length is 4 because "😂" takes up two code units, while the total number of actual Unicode characters is 3 (A, 😂, and B).

Why Java Uses UTF-16 Instead of UTF-8

UTF-8 is the dominant encoding in many modern applications, but Java chose UTF-16 for its String class for historical reasons. When Java was first developed in the 1990s, Unicode characters were initially defined within a 16-bit space (U+0000 to U+FFFF), making UTF-16 seem like a good fit.

That being said, as Unicode expanded to support over a million possible characters, UTF-16 became a variable-length encoding, requiring surrogate pairs for any character beyond U+FFFF. This made some operations, like indexing characters in a string, more complex compared to a truly fixed-width encoding.

Despite this, Java continues using UTF-16 internally because:

1. Many characters still fit within a single 16-bit unit, making operations like slicing and indexing efficient for most common cases.
2. Historically, UTF-16 could be more space-efficient for languages like Chinese or Japanese, since each character in the Basic Multilingual Plane uses 2 bytes instead of the 3 bytes often needed in UTF-8. But, modern systems frequently favor UTF-8 for broader interoperability and more efficient storage when text includes mostly ASCII characters.
3. UTF-16 avoids some of the complexities of UTF-8 when working with fixed-width character storage, such as in certain text processing applications.

Working with UTF-16 Strings in Java

Since Java’s String class operates on UTF-16 code units, it’s important to handle multi-unit characters properly. Using methods like codePoints() allows processing strings while respecting full Unicode characters.

Example:

public class CodePointsExample {
    public static void main(String[] args) {
        String text = "Hello 🌍"; // Contains a globe emoji (U+1F30D)

        text.codePoints().forEach(cp -> 
            System.out.println("Code point: " + Integer.toHexString(cp))
        );
    }
}

Output:

Code point: 68  
Code point: 65  
Code point: 6c  
Code point: 6c  
Code point: 6f  
Code point: 1f30d

Here, Java correctly processes 🌍 as a single U+1F30D code point, even though it takes up two UTF-16 code units in memory.

Encoding Mismatches and Display Issues

Java’s UTF-16 encoding allows for a wide range of characters, but problems arise when text is transferred between systems or applications that expect a different encoding. These mismatches can lead to unreadable characters, missing text, or symbols that appear as gibberish. Understanding why this happens requires looking at how encoding works at the byte level and how different systems interpret character data.

How Encoding Mismatches Happen

Text files, network communication, and database storage all rely on encoding to interpret byte sequences as readable characters. If an application saves text using UTF-16 but another system reads it as UTF-8 or another encoding, the byte sequences may be misinterpreted.

For example, if a Java program writes a file in UTF-16, each character is stored using at least two bytes. However, if another application reads the file as UTF-8, it will not recognize the two-byte sequences correctly, leading to display errors.

A common example involves the smiley face character (U+263A):

• Stored in UTF-16:

263A (Hex) -> 00100110 00111010 (Binary)

• Interpreted incorrectly as UTF-8:

☺

Instead of showing ☺, the system reads the raw bytes as separate characters, resulting in ☺ instead of the intended output.

Another common mistake happens when data is stored in a database using one encoding but retrieved using another. For instance, a database configured for ISO-8859–1 may store a name correctly, but a Java application reading it as UTF-8 may display incorrect symbols or even lose characters entirely.

Detecting and Preventing Encoding Mismatches

One way to detect an encoding mismatch is to check for unexpected sequences of characters in output, such as � (the replacement character) or garbled symbols where text should appear. If a file or data stream appears unreadable, checking the byte representation can help determine how the text was originally encoded.

Java provides tools for handling encoding properly when reading and writing data.

Reading a File with the Correct Encoding

import java.nio.file.*;
import java.io.IOException;
import java.nio.charset.StandardCharsets;

public class ReadFileExample {
    public static void main(String[] args) throws IOException {
        String content = Files.readString(Path.of("example.txt"), StandardCharsets.UTF_8);
        System.out.println(content);
    }
}

Here, StandardCharsets.UTF_8 makes sure Java interprets the file as UTF-8 instead of the platform’s default encoding. If the file was saved in UTF-16, StandardCharsets.UTF_16 should be used instead.

Writing a File with a Specific Encoding

import java.nio.file.*;
import java.io.IOException;
import java.nio.charset.StandardCharsets;

public class WriteFileExample {
    public static void main(String[] args) throws IOException {
        String content = "Java encoding test";
        Files.writeString(Path.of("output.txt"), content, StandardCharsets.UTF_8);
    }
}

Specifying StandardCharsets.UTF_8 makes sure that any application reading the file will recognize it correctly.

Character Corruption in Network Communication

Encoding mismatches can also occur when sending text over a network, especially when working with HTTP requests or socket communication. If the sender and receiver do not agree on the encoding, data may be corrupted in transit.

For example, sending UTF-16 text to a system expecting UTF-8 can cause multi-byte characters to be misinterpreted. This is why HTTP headers and request bodies should always explicitly specify the encoding.

Example of Sending Data with a Defined Encoding

import java.net.http.*;
import java.net.URI;
import java.nio.charset.StandardCharsets;
import java.io.IOException;

public class HttpRequestExample {
    public static void main(String[] args) throws IOException, InterruptedException {
        HttpClient client = HttpClient.newHttpClient();
        HttpRequest request = HttpRequest.newBuilder()
                .uri(URI.create("https://example.com"))
                .header("Content-Type", "text/plain; charset=UTF-8")
                .POST(HttpRequest.BodyPublishers.ofString("Hello, 世界", StandardCharsets.UTF_8))
                .build();

        HttpResponse<String> response = client.send(request, HttpResponse.BodyHandlers.ofString(StandardCharsets.UTF_8));
        System.out.println(response.body());
    }
}

Here, the request explicitly defines UTF-8 encoding, making sure that both the sender and receiver interpret the data correctly. Without this, the receiving system may assume a different encoding, resulting in unreadable text.

Database Encoding Problems

When working with databases, encoding issues can arise if Java interacts with a system using a different default character set. If a database is configured for Latin-1 (ISO-8859–1) but Java expects UTF-8, some characters may be replaced with ? or omitted entirely.

One way to prevent this is to always specify the encoding in database connections.

Example using MySQL:

import java.sql.*;

public class DatabaseEncodingExample {
    public static void main(String[] args) {
        String url = "jdbc:mysql://localhost:3306/testdb?useUnicode=true&characterEncoding=UTF-8";
        String user = "root";
        String password = "password";

        try (Connection conn = DriverManager.getConnection(url, user, password);
             Statement stmt = conn.createStatement()) {

            ResultSet rs = stmt.executeQuery("SELECT name FROM users");
            while (rs.next()) {
                System.out.println(rs.getString("name"));
            }
        } catch (SQLException e) {
            e.printStackTrace();
        }
    }
}

Adding useUnicode=true&characterEncoding=UTF-8 to the connection string tells Java to read and write text as UTF-8, preventing common encoding errors.

Encoding Issues in Java Strings

Even within Java itself, encoding mismatches can happen when converting between byte arrays and strings. If a string is stored as bytes using one encoding but reconstructed using another, the result may not match the original text.

Example of incorrect byte-to-string conversion:

import java.nio.charset.Charset;
import java.nio.charset.StandardCharsets;

public class EncodingMismatchExample {
    public static void main(String[] args) {
        String original = "Hello, 世界";

        // Encode as UTF-16
        byte[] bytes = original.getBytes(StandardCharsets.UTF_16);

        // Decode incorrectly as UTF-8
        String decoded = new String(bytes, StandardCharsets.UTF_8);

        System.out.println(decoded); // Output may be garbled
    }
}

To avoid this, always decode text using the same encoding that was used to encode it.

Corrected version:

String decoded = new String(bytes, StandardCharsets.UTF_16);

This makes sure the byte sequence is interpreted correctly, preserving the original text.

How to Handle Encoding Differences Across Systems

• Always specify the character encoding when reading or writing files to prevent misinterpretation.
• Use explicit encoding settings in databases and network requests to avoid mismatches.
• When converting between byte[] and String, always use the same encoding on both sides.
• If working with external systems, check their encoding requirements and configure Java to match.

Conclusion

Java’s use of UTF-16 encoding makes it possible to handle a wide range of characters, but working with multi-unit characters and different encoding formats requires attention to detail. Knowing how Java represents text at the byte level helps avoid problems like garbled characters or missing symbols when moving data between systems. By keeping encoding consistent when reading, writing, and processing text, developers can prevent many common issues and make sure their applications handle characters correctly across different environments.

1. *Java String Documentation*
2. *Unicode Standard*
3. *UTF-16 Encoding — Wikipedia*
4. *Java Charset and Encoding Guide*
5. *Java File Handling and Character Encoding*

Thank you for reading! If you find this article helpful, please consider highlighting, clapping, responding or connecting with me on Twitter/X as it’s very appreciated and helps keeps content like this free!