From A to 0,1
UTF-8 Everywhere Manifesto https://utf8everywhere.org/
What Every Programmer Absolutely, Positively Needs to Know About Encodings and Character Sets to Work With Text If you are dealing with text in a computer, you need to know about encodings. https://kunststube.net/encoding/
Unicode, UTF8 & Character Sets: The Ultimate Guide — Smashing Magazine This article relies heavily on numbers and aims to provide an understanding of character sets, Unicode, UTF-8 and the various problems that can arise. https://www.smashingmagazine.com/2012/06/all-about-unicode-utf8-character-sets/
The Absolute Minimum Every Software Developer Absolutely, Positively Must Know About Unicode and Character Sets (No Excuses!) Ever wonder about that mysterious Content-Type tag? https://www.joelonsoftware.com/2003/10/08/the-absolute-minimum-every-software-developer-absolutely-positively-must-know-about-unicode-and-character-sets-no-excuses/
Strings, bytes, runes and characters in Go - The Go Programming Language How strings work in Go, and how to use them. https://go.dev/blog/strings
As we’re well aware, the language of computers is spoken in electrical signals, and every piece of data we encounter daily is translated to a sequence of 0s and 1s, commonly known as binary representation.
This binary representation is applied whether it’s dealing with primitive data types like booleans, numbers, and characters, or handling extensive datasets like images, music, and videos.
When it comes to simple data types like booleans. We can store them in 1 bit, using 1 to represent true and 0 for false.
Numeric values are more complex than booleans. various numeral systems (such as octal, decimal, and hexadecimal) can be converted into binary. Additionally, handling negative values involves saving a bit for the sign, and depending on the size of your number, you can adjust the bits allocated for the number to use 8, 16, 32, or 64 bits.
However, characters are more challenging. It’s not all ASCII, right? After some research on the topic, I’ll share my findings. Maybe it’ll be useful for someone else, or it might come in handy for me if I ever need a quick refresher! 😄
Character Encoding
A quick note to mention, a string is a collection of characters. Even in certain languages like Go, it’s a series of bytes, not characters. We will come to the difference later. The key point to remember is that the fundamental unit is a character, and a string is a collection of these characters. How these characters are managed can vary from one programming language to another. We’ll delve into the differences later.
ASCII
The most practical method was to represent a character as a numeric value. This numeric value was then converted into binary form to be stored on storage drives. When reading the data, the process was reversed by a parser that understood the rules, enabling us to retrieve the original character. and that is what people at IBM thought of in 1969
By reserving about 7 bits, we can store up to 128 different characters. This capacity is more than sufficient for English letters (both lower and upper case), digits, punctuation, and even some control characters. Control characters are non-printable characters used for controlling texts, such as Backspace (BS), Horizontal Tab (HT), and Carriage Return (CR).
This is how we arrive at the ASCII table.
One question I had was about the unusual ordering of letters. I found a useful answer on Stack Overflow, which explained that it facilitates easier conversion.
For numbers, ‘110010’ represents the number ‘2’. If you remove the first two bits, you get ‘10’, the binary form of the number two, and so on.
For uppercase letters, ‘1000001’ represents ‘A’. If you remove the first bit, you get ‘1’, the index of A in alphabetical order.
Moreover, by flipping the second bit of ‘1100001’, you get the value of the lowercase ‘a’. This makes it easier to convert between cases.
But that only applies to texts written in English. English wasn’t the only language used on computers in the early days, right? So, how were non-English letters written?
Code Pages
Remember how ASCII characters can be stored in just 7 bits? Typically, the unit ‘byte’, a series of 8 bits, is used to store meaningful data. By adding an extra bit, we can store an additional 128 characters. From this, we understand that a character equates to a byte.
Each language uses these extra 128 possibilities to store its alphabet. This was sufficient for European languages, Arabic, and others. Each language has a ‘code page’, reserving the first 128 spots for ASCII characters and the rest for the language’s unique characters.
Here are examples of two code pages:
CP-1252 for European accents
CP-1256 for Arabic Abjadya
However, you need to configure the parser to know which code page to use.
This system worked well for most of the world, but the alphabets of some Asian languages were too large to fit into one byte. As a result, some systems invented double-byte characters. However, this was not a universal solution. Also with the rise of the internet and the accompanying internationalization, a system capable of interpreting all possible languages became necessary.
Unicode and UTF-*
as ASCII, the Unicode system assigned a code for each character called a ‘code point’ but with another way of thinking based on this numeric value of the code point it can be stored in a space of up to 4 bytes.
The Unicode code space is divided into seventeen planes (the basic multilingual plane, and 16 supplementary planes), each with 65,536 (= 216) code points. Thus the total size of the Unicode code space is 17 × 65,536 = 1,114,112. and that was enough to contain all the spoken alphabets also the emojis and every glyph can be drawn on the screen.
The code point for each character is written in the form U+xxxx, where ‘xx’ is a hexadecimal value that can be stored in a byte. For example, ‘A’ is U+0041, and ‘ب’ is U+0628. You can view the full table at symbl
.
In the Unicode encoding system, the number of bytes used to store the character is determined by the encoding schema. These schemas include UTF-32, UTF-16, and UTF-8, which is the most well-known.
Does the choice of schema make a difference? Using UTF-32 will always store the character in 32 bits (4 bytes), while UTF-16 will reserve only 2 bytes, using the other two only when necessary.
UTF-8
It’s not just about how well UTF-8 handles storage; it’s also about its compatibility. UTF-8 not only allocates storage byte by byte according to the character’s requirements but also maintains backward compatibility with the ASCII encoding system. In ASCII, each character is stored in one byte, and with UTF-8, a parser expects, by default, that one byte represents a character. This feature has made UTF-8 the go-to system on the internet, capable of interpreting alphabets from all spoken languages.
That leads me to two questions.
How the UTF-8 parser will know if the character is a one-, two-, three-, or four-byte character?
- For multibyte encodings, the first byte sets 1 in a number of high-order bits equal to the number of bytes used in the encoding; the bit after that is set to 0. so a 2-byte sequence always starts with 110 in the first byte
- For all subsequent bytes in a multibyte encoding, the first two bits are 10
And this also partially answers the next question. since we have 4 bytes (32 bits) to store on doesn’t this give us a total of (2^32) = about 2B characters?
and as it appears it is not all 32 bits are used for storing the code point, there is bits to inform the encoding schema about multibyte characters.
Golang perspective
- Golang has no data type for characters. It uses runes and bytes to represent a character value.
- A string is a read-only slice of bytes, so indexing a string yields its bytes.
- Use for range loop, it decodes one UTF-8-encoding rune for each iteration,
- For further dealing with runes on golang you can check the packages unicode and unicode/utf8
At the end
I hope you found this article helpful, and if there’s anything I overlooked, please don’t hesitate to correct me.
Also, here’s a handy list of resources I stumbled upon during my research: