Debugging your compiler
A red test is information. This page is the routine: read what the grader says, find out what the test is for, shrink it until the bug has nowhere to hide, and compare with the oracle.
1. Read the grader's output
Run the suite with -v so every failure prints its details, and -k to run only the tests whose path contains a word:
python tools/test.py --compiler YOUR_COMPILER --week 1 -v
python tools/test.py --compiler YOUR_COMPILER --week 1 -k div -vYOUR_COMPILER is the command that runs your compiler. For a Python compiler that's the wrapper script from lesson 0.7, such as "C:/path/to/my-ml-compiler/mlc.cmd" on Windows or ~/my-ml-compiler/mlc on macOS and Linux.For an OCaml compiler it's the path to the executable dune built, such as C:/path/to/_build/default/bin/main.exe.
Each failure is a FAIL line with the test's path, then one or more indented lines saying what was wrong. These are real messages from the grader:
FAIL tests/week1/run/i32_div_trunc.uir
expected exit 0 (pass), got 1
stderr: assert failed: %t0Your compiler said an assert failed, but the oracle says every assert passes. So one of your values is wrong, and %t0 is the first assert that saw it.
FAIL tests/week1/run/err_div_zero.uir
exit 2 is right, but stderr needs a line starting 'invalid:' (an uncaught exception isn't a diagnosis)Your compiler exited 2, but by crashing. OCaml exits 2 on an uncaught exception, so the grader also requires an invalid: line on stderr. Catch your errors and print the reason. In Python an uncaught exception exits 1 instead, so it shows up as a failed assert with no assert failed: line. The fix is the same: catch it, print invalid: and the reason, and exit 2.
The run contract in one table
| exit | meaning | stderr must contain |
|---|---|---|
| 0 | every assert passed | anything |
| 1 | an assert failed | exactly one line assert failed: %name, naming the first failing assert in file order |
| 2 | invalid program: parse error, type error, or a runtime error an assert depends on | a line starting with invalid: |
Other messages you'll see, and what they point at:
| message starts with | look at |
|---|---|
expected exit 2 (invalid), got 0 | a check you don't do yet: a type rule, a bounds proof, an ordering rule. The oracle: line under it says which. |
N ops in your output, the target is <= M | a simplification you're missing, or duplicate statements in your output (hash cons what you print) |
wrong value: original gives ..., yours gives ... | a rewrite that isn't true for every input. The args: line is the input that breaks it. |
rendered C is N bytes; the limit is 20000 | a vector you unrolled instead of looping over |
your C doesn't compile | the compiler's error is printed right below, so check names, types and missing semicolons in your printer |
your kernel crashed | undefined behavior caught by the sanitizer (see section 5) |
buffer 2[3]: expected ..., your kernel left ... | the first wrong output element: which buffer slot, which index |
2. Find out what the test is for
Every test starts with a comment saying what it checks. The test guides collect all of them, with the plausible bugs known to fail each test: week 1, week 2, week 3, week 4, week 5. Read the matching section of the spec for that week before changing code: most failures are a rule read differently, not a typo.
3. Shrink the failing test
A test with ten statements has ten places for the bug. Copy it to a scratch file and delete lines until it stops failing, then put back the last thing you removed. What's left is the bug.
Here's i32_div_trunc.uir, which failed above:
%a = div -7 2
%t0 = assert %a -3
%b = div 7 -2
%t1 = assert %b -3
%c = div -7 -2
%t2 = assert %c 3
%d = div 7 2
%t3 = assert %d 3The failure named %t0, so keep only what %t0 needs:
%a = div -7 2
%t0 = assert %a -3Ask the oracle what should happen, then try your own guess for the value:
> python _reference/uirc.py run small.uir
(exit 0: -7 div 2 really is -3)
> python _reference/uirc.py run guess.uir # the same program asserting -4
assert failed: %t0
(exit 1)Your compiler computes -4: it rounds toward minus infinity, like Python's // or a floor on floats. The spec says division truncates toward zero, like C. Two lines, one rule.
- Delete asserts after the failing one first. They never run.
- Replace a computed operand with the constant it should be. If the bug disappears, the bug is upstream.
- For simplify failures, put the grader's
args:values in by hand and compare both programs. - Keep a folder of shrunk cases. They make good tests for your own suite.
4. Compare with the oracle
The oracle is the reference compiler. Running it is fine and encouraged, but reading its source would hand you the answers.
python _reference/uirc.py run FILE # exit code and assert line, exactly what the grader expects
python _reference/uirc.py simplify FILE # one valid simplification of %out (not the only one)
python _reference/uirc.py render FILE # the oracle's C for a week 2+ programAn example: this program should simplify to its input, and the oracle shows it does.
%x = param 0 dtype=i32
%a = add %x 0
%out = mul %a 1
> python _reference/uirc.py simplify s.uir
%out = param 0 dtype=i32Your output doesn't have to match the oracle's text. It has to compute the same values and stay within the op limit. For render, compare your C's behavior, not its look: the oracle's C shows one correct schedule, and yours can differ.
5. When render fails
The grader compiles your C with these flags, then runs it on the test's data and compares every buffer bit for bit:
clang -std=c11 -O2 -ffp-contract=off -fsanitize=undefined -fsanitize-trap=undefined| symptom | usual cause | fix |
|---|---|---|
| your C doesn't compile | a name used before it's declared, a type printed wrong, a missing include | save your output to a file and compile it yourself with the flags above to read the full error |
| your kernel crashed (exit code nonzero) | signed overflow, INT_MIN / -1, a shift outside 0..31: the sanitizer turns undefined behavior into a crash | wrap i32 add, mul and shl through uint32_t, and guard div and mod against -1 |
| didn't finish in time | on Windows a plain INT_MIN / -1 can hang instead of trapping | the same -1 guard |
| rendered C over 20,000 bytes | unrolled vectors | emit a for loop per rank-1 store and per reduce |
| f32 off in the last bits | a decimal literal that doesn't round-trip, the wrong fold order, or a * b + c fused into one rounding | print f32 constants as hex floats (%a style, 0x1.8p+1f), and fold reduces left to right from the first element |
| wrong element, right neighbors | an off-by-one in an index or loop bound | check < vs <= and where offsets add |
To see exactly which bits differ, print a float's bits in your own test harness:
#include <stdint.h>
#include <string.h>
uint32_t bits_of(float x) { uint32_t u; memcpy(&u, &x, 4); return u; }
/* printf("%08x\n", bits_of(v)); */3e99999a and 3e99999b differ in the last bit: one rounding step somewhere.
6. Print your IR between passes
Most bugs live between two passes: the graph was right after parsing and wrong after simplifying, or right before rangeify and wrong after. Give your compiler a debug flag that prints the graph as .uir after each pass, then diff the printouts. Printing .uir has a bonus: you can feed any intermediate stage back to the oracle and ask what it computes.
7. Usual suspects, week by week
These are mistakes planted into the reference compiler to prove the tests catch them. If a test below fails, start with the advice next to it.
| week | check that | tests that catch it |
|---|---|---|
| 1 | div truncates toward zero, and mod takes the sign of the left operand | i32_div_trunc, i32_mod_sign |
| 1 | every f32 op rounds to f32, not just the final result | f32_rounding_step, f32_product_rounds, f32_overflow |
| 1 | you cache each node's value, so shared work runs once | dag_sharing_300, dag_sharing_mix |
| 1 | the cache is per call, not global | call_memo_per_call, call_composition |
| 1 | a nested call's body only sees its own args | call_closed_body |
| 1 | asserts run in file order, and an assert compares NaN equal to NaN | fail_first_in_order, f32_assert_equality |
| 1 | x * 0.0 isn't 0.0, and (x * 2) / 2 isn't x | trap_mul_zero_f32, trap_div_mul_wrap |
| 2 | offsets of a view of a view add up | view_shrink_of_shrink |
| 2 | loads of a version run before the store that overwrites it | swap_schedule, old_version_late_use |
| 2 | a view's bounds are proven, including the last element | err_bounds_past_end, err_bounds_mask_too_wide |
| 2 | your C wraps i32 through uint32_t and guards div/mod by -1 | i32_add_wrap, i32_div_mod_intmin |
| 2 | f32 constants print exactly (hex floats) | f32_exact_literals |
| 3 | an accumulator's load moves into the loop that writes it | r_conv1d_valid, r_conv1d_padded |
| 3 | a loop runs exactly n times, and zero times for n <= 0 | bounds_from_ranges, zero_trip, negative_trip |
| 3 | an after inside a loop isn't a buffer's output | inloop_after_not_output |
| 4 | a reduce folds left to right, starting from the first element (not from 0) | reduce_add_f32_order, reduce_add_negzero |
| 4 | vectors become loops in render, even a plain 4096-element copy | normalize_4096, r_copy_4096 |
| 5 | reshape is row-major (the last axis changes fastest) | broadcast_column_sums_back, expand_rank3_middle |
| 5 | permute applies the order you were given, not its inverse | permute_rank3_all_orders, permute_inverse_roundtrip |
| 5 | pads are (left, right) per axis, and the mask is exact at both edges | pad_2d_mixed, pad_2d, r_conv2d_16x16 |
8. Asking for help well
Whoever helps you, a classmate or an AI assistant, needs the same four things:
- The smallest program that fails (section 3), not the whole test.
- What you expected, what you got, and what the oracle says.
- The rule from the spec you think applies.
- What you already tried.
The AGENTS.md from templates/ sets the rule for your compiler repo: an AI assistant may explain concepts, draw diagrams, run the oracle, and ask you questions that lead to the bug, but it doesn't write or edit your compiler code. Ask "why would this give -4?" rather than "fix my div".
9. When the test itself looks wrong
It happens. The format is provisional and the tests are written by people. Check in this order:
- Read the spec section again, slowly. Most "wrong tests" are a rule that says something subtler than it first seemed.
- Run the oracle on the test. The grader compares you to the oracle, not to the header, so the oracle's answer is what you're held to.
- If the spec and the oracle disagree, that's a real bug in the course. Write it up with the smallest program that shows it, the spec sentence, and what the oracle does, and report it to whoever maintains your copy of the course.