dtypes of tensors: bfloat16 vs float32 vs float16

You may have come across the term bfloat16 in the context of machine learning and artificial intelligence. This article explains what bfloat16 is and compares it with other floating-point formats.

bfloat16, known as “brain floating-point 16-bit”, is a computer floating point format occupying 16 bits in computer memory. It was developed by Google Brain with the intent of reducing the storage requirements and accelerating computation in machine learning.

bfloat16 is a shortened version of the 32-bit IEEE 754 single-precision floating-point format (float32). It preserves the dynamic range of float32 numbers by retaining 8 exponent bits and allows for fast conversion to and from a float32 number. While bfloat16 uses the same number of bits as float16, it has a wider dynamic range but lower precision. More quantitative comparisons follow.

Let’s do a review of the IEEE 754 floating-point format. Then we compute the minimum and maximum numbers the different formats can represent.

Floating-point formats (adapted from https://cloud.google.com/tpu/docs/bfloat16)

bfloat16

bfloat16 is a custom 16-bit floating point format that is composed of one sign bit (S), 8 exponent bits (E), and 8 mantissa (also known as significand or fraction) bits (M). For mantissa bits, 7 bits are explicitly stored, with an implicit leading bit. bfloat16 uses 1 + 8 + (8–1) = 16 bits, as shown in the above figure. Each bit can take up a value of either 0 or 1 (binary). A 16-bit floating point format can, in principle, represent 2¹⁶ = 65536 distinct numbers, but in practice less numbers are represented due to the multiple representations of NaNs (see Not a number NaN section below).

float32 and float16

float32 is the single-precision floating-point format occupying 32 bits in computer memory whereas float16 is the half-precision floating-point format.

Sign bit

If the sign bit is 0, the number is positive. If the sign bit is 1, then the number is negative.

Zeros

There are two zeros: 0 and -0. The number is 0 when all bits are 0, and -0 when all bits are 0 except the sign bit.

Positive and negative infinity ±∞

The number is infinity ∞ when all exponent bits are 1 and all mantissa bits are 0. +∞ in bfloat16 is represented as 0-11111111-0000000 whereas -∞ is 1-11111111-0000000.

Not a number NaN

NaN values are represented with either sign bit, all exponent bits 1 and not all mantissa bits zero. Therefore, multiple representations are reserved for NaN. There can be three separate kinds of NaNs: quiet NaN (qNaN), signaling NaN (sNaN) and optional diagnostic information (sometimes called a payload). qNaNs are used to propagate errors resulting from invalid operations or values. sNaNs can support advanced features such as mixing numerical and symbolic computation or other extensions to basic floating-point arithmetic.

In bfloat16, there are 2⁸- 2 = 254 NaN representations. The number 8 comes from one sign bit and 7 mantissa bits. Minus 2 is for the two infinity representations.

Exponent bits

The all-one exponent is interpreted specially as either infinity or NaNs as stated above. The following sections consider two cases: (1) all-zero exponent (subnormal numbers) and (2) the rest that are not all-zero and not all-one exponent (normal numbers).

The exponent is encoded using an offset-binary representation, with the zero-offset (also known as exponent bias) being 2^(n-1)-1 where n is the number of bits in exponent. With this definition of zero-offset, about half of the numbers is larger than 1 and the other half smaller than 1. In bfloat16, the zero-offset is 01111111 in binary or 2⁷-1 = 127 in decimal. For (2) not all-zero and not all-one exponent, with this offset, the smallest exponent is 1-127 = -126 since the smallest binary number for 8-bit is 00000001 when all-zero is excluded, and the largest exponent is (2⁸-2)-127 = 127 when the largest binary number for 8-bit is 11111110 when all-one is excluded.

Mantissa bits

The mantissa is also known as fraction or significand. The mantissa has one implicit leading bit and multiple explicit bits. For (1) all-zero exponent, the implicit leading bit of the mantissa is 0. Whereas for (2) not all-zero and not all-one exponent, the implicit leading bit of the mantissa is 1. When the implicit leading bit is 1, it contributes 1/2⁰ = 1 to the mantissa. As for the explicit bits, the leftmost bit contributes 1/2¹ = 0.5 to the mantissa, the second leftmost bit contributes 1/2² = 0.25,…, and the i-th bit contributes 1/(2^i), if that bit is 1. The value of the mantissa is the sum of the contribution of all implicit and explicit bits that are 1.

Floating-point numbers

Let s ∈ {0,1} be the sign bit, n be the number of bits in the exponent, E ∈ {1,2,…,2^n-2} be the value of the exponent in decimal, Z be the zero-offset in the exponent and M be the value of the explicit mantissa, which is the sum of the contribution of all explicit mantissa bits that are 1.

For (1) all-zero exponent, the floating-point format represents the number (-1)^s × 2^(1-Z) × M. Whereas for (2) not all-zero and not all-one exponent, numbers are (-1)^s × 2^(E-Z) × (1+M).

The above number representation is general for IEEE 754 floating-point format. Let’s look at some bfloat16 examples.

Example 1: 0-00000000-0000010

M = 1/2⁶

number = (-1)⁰ × 2^(1-127) × 1/2⁶ = 1/2¹³²

Example 2: 1-10000010-1010000

M = 1/2 + 1/8 = 5/8

number = (-1)¹ × 2^(130-127) × (1+5/8) = -2³ × 13/8 = -13

Example 3: the minimum positive normal number

0-00000001-0000000 is (-1)⁰ × 2^(1-127) × (1 + 0) = 1/2¹²⁶ ≈ 1/10³⁸

Note: A normal number is a floating point number when the exponent bits are not all-zero and not all-one.

Example 4: the minimum positive subnormal number

0-00000000-0000001 is (-1)⁰ × 2^(1-127) × 1/2⁷ = 1/2¹³³ ≈ 9/10⁴¹

Note: If all exponent bits are zero, the implicit leading bit is 0. The represented numbers are subnormal. Subnormal numbers are smaller in magnitude than the smallest normal number and fill the underflow gap around zero in floating-point arithmetic (exercise: find the maximum positive subnormal number!).

Example 5: the maximum positive number

0-11111110-1111111 is (-1)⁰ × 2^(254 -127) × (1+127/128)

= 255 × 2¹²⁰ ≈ 2¹²⁸ ≈ 3 × 10³⁸

Range, minimum and maximum: bfloat16 vs float32 vs float16

The minimum and maximum for float32 and float16 can be computed in the same way as above for bfloat16.

Precision: bfloat16 vs float32 vs float16

Step size in different intervals for bfloat16, float32 and float16 formats

Precision can be characterized by step size, or the difference between two consecutive numbers, in different intervals. The smaller the step size, the higher is the precision.

As we can see, bfloat16 has the same dynamic range as float32, and the range is much wider than float16, at the expense of precision.