An introduction to Unicode and why it’s important

It is possible that one of the most important strides into making the internet accessible to all was the development of Unicode, the de facto standard for encoding text in nearly all languages. When computers first were made able to process text, only letters from the English alphabet were considered. Later on, support for letters with accents and other diacritics was added, allowing text to be processed in many European-based languages (those with alphabets derived from Latin). However, there remained many languages that computers were unable to work with. These included languages with different alphabets and scripts, and in the case of Chinese, Japanese, and Korean (abbreviated CJK), there were orders of magnitude more characters. As with much technological development, many different solutions arose at the same time to address this problem, many of which were not mutually interoperable. Particular confusion arose in CJK, as many more characters needed to be considered, and there were many means of encoding each language. Over time, this caused difficulty not only because a user’s computer had to support many different languages separately, but also because even within the same language there were many different encodings. Eventually the problem became serious enough that a standard, Unicode, was developed. The Unicode standard defines a hexadecimal number for nearly every possible character used in modern languages, up to and including CJK characters. Various systems are now used to convert those characters into a binary format, a process known as encoding. The most popular one is known as UTF-8. This format and the Unicode standard solved a major problem in the internationalization of modern communication.

For new developers starting out, this may seem unimportant. After all, nearly everyone uses it, the internal details are rarely exposed, and no knowledge is needed to use it except how to type in a given language. However, this assumption fails when working with older documents or fonts that do not supposed the desired character set. For example, a file written in ASCII, an early encoding, that uses characters in certain European languages (such as accented vowels) block will not render correctly if decoded as UTF-8. A UTF-8 encoded file that contains characters not in the basic Latin block will display incorrectly if decoded as ASCII. In addition, it is important to know what fonts are supported by a font before using the font. If you need a font supporting the CJK characters, for example, you cannot use Times New Roman. Rather, you must use a font that supports those characters. For anyone working in non-English languages, a basic understanding of Unicode is very valuable.