FP16, BF16, FP8 and FP4 converter

The formats side by side

Every one of these is the same idea as the 32 and 64 bit floats on the IEEE 754 converter: a sign bit, an exponent stored with a bias, and the bits after the point of a binary number 1.mmm. They differ only in how many bits go to the exponent, which sets the range, and how many to the mantissa, which sets the precision. Every figure in this table is computed by the converter's engine.

sign exponent mantissa, drawn to scale. BF16's exponent is exactly as wide as FP32's.

Epsilon is the gap between 1 and the next value up, 2 to the minus number of mantissa bits. Digits is the significand's bits, hidden 1 included, times log₁₀ 2: roughly how many significant decimal digits survive.

How the bits make a value

For a normal number, the value is (−1)^sign × 1.mantissa × 2^{exponent − bias}. The 1 before the point is not stored, because a normal number always has it. When the exponent field is all zeros the number is subnormal: the hidden bit becomes 0 and the exponent stays at 1 − bias, so the values fade evenly down to zero instead of stopping short of it.

The formats disagree about the all-ones exponent. The IEEE-style ones (FP32, FP16, BF16 and E5M2) reserve it: mantissa zero is infinity, anything else NaN. E4M3 keeps only one NaN, S.1111.111, and uses the rest of the top exponent for ordinary numbers, which is how it reaches 448 instead of 240. LLVM and PyTorch call it E4M3FN: F for finite, N for its non-IEEE NaN. FP4 E2M1 reserves nothing at all: every code is a number.

Rounding: to nearest, ties to even

A decimal almost never lands on a value the format can store, so it is rounded to the nearest one. When it is exactly halfway between two, it goes to the one whose last mantissa bit is 0, the even one. Always rounding halves up would push sums upwards on average; ties to even goes up half the time and down half the time.

Hardware does this with three extra bits. Beyond the bits it keeps, it remembers the next bit (the guard), the one after that (the round bit) and whether anything at all is set further down (the sticky bit, an OR of the rest). Guard 0 means less than half a step: truncate. Guard 1 with round or sticky set means more than half: round up. Guard 1 with both clear is an exact tie, and the last kept bit decides.

Too big: infinity, saturation or NaN

Rounding works as if the exponent could keep growing, and only then checks whether the result fits. So the cut-off is not the largest value itself but halfway to the step above it. In FP16 that is 65520: exactly halfway, and the even neighbour is the pattern above 65504, infinity. In E4M3 it is 464, halfway between 448 and the 480 that S.1111.111 would have been if it were not NaN; here the even neighbour is 448, so 464 stays finite and 465 does not.

Formats with infinity overflow to it. E4M3 and FP4 have none. For E4M3 the OCP 8-bit floating point specification allows either saturating to ±448 or producing NaN, and it also lets conversions to E5M2 saturate instead of overflowing. This converter shows: infinity for FP32, FP16, BF16 and E5M2; for E4M3 whichever of the two you pick above (saturation unless you change it); and for FP4, which has neither infinity nor NaN, the largest value, ±6.

Why BF16 keeps FP32's range

BF16 is the top 16 bits of an FP32: the same sign, the same 8 bit exponent with the same bias of 127, and the first 7 of FP32's 23 mantissa bits. So BF16 covers almost the same range as FP32, only less precisely: its largest value is 3.3895e38 against FP32's 3.4028e38, and its subnormals stop at 2⁻¹³³ rather than 2⁻¹⁴⁹. Converting is a matter of rounding off the low 16 bits. FP16 has more precision but a 5 bit exponent, so it runs out at 65504. The same values in all three:

The BF16 pattern is the first four hex digits of the FP32 one, or one more when the low half is past halfway and rounds up, as it does for 0.1.

Block scaling: how FP4 is usable at all (MXFP4)

FP4's 7 positive values (8 with zero) are too few to store anything useful on their own. The OCP microscaling (MX) formats fix that by sharing a scale: a block of 32 values stores one 8-bit scale, an E8M0 number that is nothing but a power of two (an exponent with bias 127 and no mantissa), and each value is stored as an E2M1 after dividing by it. The scale exponent is the power of two of the block's largest magnitude minus 2, the largest exponent of E2M1, so the largest value lands between 4 and 8; anything above 6 is clamped.

Here is a block of 8 values (a real block has 32), worked by the engine. The largest magnitude sets the scale to 2⁻¹, stored as E8M0 code 126:

The block costs 4 bits per value plus 8 bits shared by all 32, 4.25 bits per value. Small values in a block with a large one lose the most: they fall into E2M1's coarse steps near zero.

Common mistakes

• Rounding twice. Converting a decimal to FP32 and then to BF16 is not always the same as rounding it straight to BF16. 1.003906250931322574615478515625 (that is 1 + 2⁻⁸ + 2⁻³⁰) goes straight to 1.0078125 (3F81), but FP32 first rounds it to 1.00390625, an exact tie for BF16, which then goes to 1 (3F80). This converter always rounds once, from the exact decimal.
• Chopping instead of rounding. Keeping the top 16 bits of an FP32 truncates it. 3DCCCCCD (0.1 as FP32) chopped is 3DCC = 0.099609375; rounded properly it is 3DCD = 0.10009765625.
• Assuming FP16 has room. Its largest value is 65504, so 70000 is already infinity, and squares of numbers above about 256 overflow. BF16 or a scale factor avoids that.
• Treating E4M3 like an IEEE format. It has no infinity, 0x7F is NaN rather than infinity, and 0x78 to 0x7E are ordinary numbers from 256 to 448.
• Comparing absolute errors across sizes. E4M3 stores 100 with an error of −4 and 0.1 with an error of 0.0015625, yet relative to the value those are 4% and 1.56%, both within E4M3's bound of half its epsilon, 6.25%. For normal numbers every format here keeps the relative error within half of its epsilon; only subnormals do worse.

All 16 FP4 E2M1 values

All 256 FP8 E4M3 codes

Each row is one magnitude; the negative code is the same with the sign bit set, 0x80 higher.

For the 32 and 64 bit formats in full detail, see the IEEE 754 converter. For patterns as plain integers, the hex to binary converter and the integer limits pages.