Week 6 · Lesson 1

exp, log, sqrt and sin from adds and multiplies

A neural network needs exp and log, and your compiler only knows how to add, multiply and compare. This lesson builds four math functions out of those, the way real math libraries do: shrink the input into a small range, then use a short polynomial.

Why not just call expf?

C has expf, logf, sqrtf and sinf. But each C library computes them its own way, and two machines can disagree in the last bit. The grader compares f32 results bit for bit, so the course needs one definition everyone can match. The spec gives one: each math op is a specific program of core ops. If your compiler runs exactly those steps, your bits match the oracle's on any machine.

There's a second reason, and it's the one the syllabus cares about. A GPU, or a small accelerator, often has no exp instruction at all. Knowing how to build one from multiplies is how compilers for such hardware work.

How an f32 is stored

Every f32 is 32 bits: a sign bit, an 8-bit exponent field E, and a 23-bit mantissa M. For normal numbers the value is (-1)^sign · 1.M · 2^(E - 127). The 127 is a bias, so exponents from -126 to 127 fit in a field from 1 to 254. When the exponent field is 0, the number is subnormal: its value is (-1)^sign · 0.M · 2^-126. These are the tiny numbers below 2^-126, with fewer bits of precision.

The new op bitcast x dtype=i32 gives you those 32 bits as an integer, and bitcast n dtype=f32 goes the other way. Nothing is converted: the same bits are read as the other type. That's what lets the math ops pull a number apart and put one together.

exp2: split, then a polynomial

2^x for any x comes from two easy cases. Round x to the nearest integer n and let f = x - n, so f is in [-0.5, 0.5]. Then 2^x = 2^f · 2^n.

The spec evaluates the polynomial with Horner's rule: c0 + f(c1 + f(c2 + ...)) with c_k = (ln 2)^k / k! (1.0, 0.6931472, 0.2402265, ...), starting from the highest coefficient. That's one multiply and one add per term, and it's also more accurate than adding up powers.

Two details make the edges right. First, NaN is caught before anything else, because casting NaN to i32 is an error. Second, x is clamped to [-150, 128]. 2^128 is already infinity in f32, and 2^-150 already rounds to 0, so clamping changes no answer and keeps n small. The bit trick (n + 127) · 2^23 only works for n from -126 to 127, and after the clamp n runs from -150 to 128. Below -126 the result is subnormal, and at 128 a single 2^128 would already be infinity even when the final result (say exp2 127.9) is finite. So the spec multiplies by 2^h and then 2^k, with h + k = n and both in [-75, 64], safely normal.

2^n from its bits

Write pow2 n for -126 ≤ n ≤ 127 by building the f32 bit pattern of 2^n and reading it as a float. No exponentiation allowed.

log2: read the exponent, fix up the rest

log2 runs the same idea backward. Bitcast x to an integer b. Its exponent field is b div 2^23, so e = b div 2^23 - 127. Its mantissa bits are b mod 2^23. Adding the bits of 1.0 (1065353216) gives the f32 with that mantissa and exponent 0, which is m, in [1, 2), with x = m · 2^e. So log2 x = e + log2 m.

A series for log2 m converges fastest when m is close to 1, so if m > 1.414 the spec halves it and adds 1 to e. Now m is in [0.707, 1.414]. With s = (m - 1) / (m + 1), |s| ≤ 0.172 and

log2 m = (2 / ln 2) · (s + s³/3 + s⁵/5 + s⁷/7 + s⁹/9 + ...)

Five terms give full f32 precision. A subnormal x has exponent field 0, so it's multiplied by 2^23 first and 23 is subtracted at the end. The special cases are checked at the end: zero gives -inf, a negative gives NaN, and inf and NaN pass through.

sqrt: a guess, then Newton

sqrt x = 2^(log2(x) / 2) is a fine first guess, but two rounded steps leave it a few units in the last place (ulps) off. One ulp is the gap between an f32 and the next one. One step of Newton's method fixes that. If y is close to √x, then (y + x/y) / 2 is much closer, because the error roughly squares each step. Perfect squares come out exact: sqrt 16.0 is exactly 4.0.

sin: subtract turns, fold, then Taylor

sin repeats every 2π, so first subtract the nearest whole number of turns: k = floor(x / 2π + 0.5), then r = x - k · 2π. 2π isn't an f32, so the spec subtracts it in two parts: 6.28125, which is exact, and a small correction. That keeps r accurate for x up to about 10,000.

Now r is in about [-π, π]. Since sin(π - r) = sin r, folding brings it to [-π/2, π/2], where the Taylor series r - r³/3! + r⁵/5! - ... up to r^13 is accurate. For a huge x, f32 can't even say which turn x is on, so the result is meaningless. The spec still clamps it so it stays in [-1, 1].

Close is not equal

A better polynomial, a different reduction, or C's expf can all be more accurate than the spec, and they still fail the tests. exp2_bits_one_ulp_off asserts a value one ulp away from the right one and is marked to fail. A compiler that is one ulp off makes that assert pass, and the test catches it. Implement the steps exactly as written, in the order written. tools/check_spec_w6.py proves the spec text and the oracle are the same graph, so the spec is the thing to copy.

The length of a vector

Slot 0 holds six f32. Store sqrt(x0² + ... + x5²) into slot 1. The starter stores the sum of squares.

Special values fall out

Nobody special-cases exp2(-inf) = 0: the clamp sends it to -150, and 2^-150 rounds to 0. Most of the table in the spec works like that. The cases that need an explicit where are the ones where a step would otherwise fail or lie: NaN before a cast, zero and negatives before a log, and infinity, whose bits look like a huge exponent.

One more rule makes NaN predictable. There are millions of NaN bit patterns, and different hardware makes different ones (x86 produces a negative NaN for 0 · inf). bitcast of any NaN gives 0x7fc00000, so bitcast never exposes which NaN you had. In C, check isnan first.

Check yourself

What does bitcast 2.0 dtype=i32 give?

2.0 is 1.0 · 2^1: mantissa 0, exponent field 1 + 127.

Exponent field 128, shifted into place: 128 · 2^23 = 1073741824, which is 0x40000000.

For exp2 3.3, what are n and f?

n = floor(x + 0.5), the nearest integer.

floor(3.8) = 3, so f = 3.3 - 3, about 0.3 (exactly, the f32 nearest 3.3 minus 3). Then 2^3.3 = 2^0.3 · 8.

Why does exp2 clamp x to [-150, 128] before anything else?

What is 2^1000 in f32?

Every x above 128 gives inf and every x below -150 gives 0, so clamping is free. Without it, exp2 1e30 would cast 1e30 to i32, which is an error.

12.0 is 1.5 · 2^3. After the spec's fold, what are m and e in log2?

Is 1.5 above 1.414?

1.5 is above the fold point, so m is halved to 0.75 and e goes up to 4: 12 = 0.75 · 2^4. Then log2 12 = 4 + log2 0.75, with m close to 1 so the series converges fast.

sqrt's first guess is 2^(log2(x) / 2). Why add a Newton step?

How accurate is a result built from two approximations?

Each of exp2 and log2 is within about 1.5 units, and their errors compound. Newton's method doubles the number of correct digits per step, so one step brings sqrt back to about 1 unit.

What is log2 -0.0?

Is -0.0 below zero?

cmplt -0.0 0.0 is false: negative zero equals zero. So it isn't treated as negative (which would give nan), and the zero case gives -inf.

Your sin uses C's sinf, which is correctly rounded. What happens?

What does the grader compare against?

The oracle's result, bit for bit. A more accurate answer is a different answer. The spec defines sin as its program, not as "the true sine".