Repository navigation
Fix wrong example outputs in 1 challenge (Attention with Linear Biases) - #388
kunal-mansukhani wants to merge 1 commit into
Conversation
Row 0 of the Example 1 output listed 6.05, 7.05 in the last two columns; reference_impl gives 5.046851, 6.046850, i.e. 5.05, 6.05.
There was a problem hiding this comment.
✅ Review bot: low risk, safe to merge. Lint and structure checks passed; no sensitive paths or grader edits; 2 lines changed.
- Docs-only change: one line in challenge.html, with no challenge.py, tests, tolerances or starters touched.
- The corrected values 5.05 and 6.05 check out by hand. Softmax of [0.5,-0.5,-1] is about [0.6285,0.2312,0.1402]. Applied to V's third column [3,7,11] that gives about 5.047, matching the PR's reference_impl result of 5.046851.
- The corrected row 0 (3.05, 4.05, 5.05, 6.05) increases by 1 per column, consistent with row 0 of V being [1,2,3,4] (inferred from the output pattern).
- No new challenge is added, so the duplicate, ID and starter checks do not apply. The diff is complete and contains no suspicious content.
shxjames
left a comment
There was a problem hiding this comment.
Reviewed bb66508. The Example 1 fix is correct, and Example 2 was already right.
Example 1 matches both reference_impl and an independent fp64 reimplementation of the written formula, softmax(QKᵀ/√d + α(i−j)) V, with no causal mask. Row 0 is 3.04685, 4.04685, 5.04685, 6.04685, which rounds to the new 3.05, 4.05, 5.05, 6.05. The old 6.05 / 7.05 were off by exactly 1. generate_example_test() uses the same inputs (also functional test 2). Row 1 (3.93 … 6.93) was already correct.
Example 2 (3.95, 4.95) also matches; fp64 is 3.953587, 4.953587. Printed precision is consistently 2 decimal places. There are no other examples in this challenge.
Docs-only HTML change; tests, tolerances, and starters are untouched.
Pre-existing, non-blocking: the spec is not causal (the ALiBi paper usually is), and that agrees with the reference and both examples. atol = rtol = 1e-4 leaves correct fp32 ≥150× headroom under the max-scaled grader. Perf (2048×2048×1024) is about 34 MB I/O plus 17 MB for S and ~17 GFLOP, fine on a T4. The constraints line omits d = 1024.
CI is green. I have not approved.
|
Validated bb66508.
Non-blocking, pre-existing: the performance bullet lists only LGTM. 🤖 Generated with Claude Code |
Audit of every challenge on
main(147) for wrong example outputs inchallenge.html, the same kind of bug #331 fixed in Causal Self-Attention. One real mismatch turned up, and this PR fixes it.Fix:
medium/55_attn_w_linear_bias, Example 1reference_impl(CPU, fp32)Row 1 (
3.93 4.93 5.93 6.93) and Example 2 (3.95 4.95; reference gives 3.953587, 4.953587) were already correct.By hand: for query 0, the logits are
[1,0,0]/sqrt(4) = [0.5,0,0]and the ALiBi bias is0.5*(0-j) = [0,-0.5,-1]. Softmax of[0.5,-0.5,-1]is[0.6285, 0.2312, 0.1402]. Applied to V's third column[3,7,11]that gives 5.047, so every column is V's row-0 value + 2.047. The published 6.05/7.05 were off by exactly 1.How the audit was done
challenge.html, rebuilt its inputs, ranreference_implon CPU, and compared each displayed number at the precision it is printed. Example 1 was also checked againstgenerate_example_test().reference_implon the HTML inputs.0.244for 0.2447 ("approximately"), and 40_batch_normalization shows±1.224for ±1.2247. 20_kmeans_clustering shows no expected output, so it had nothing to compare.Flagged but not changed:
generate_example_test()doesn't mirror Example 1 in 52_silu, 14_multi_agent_sim, 61_rope_embedding and 64_weight_dequantization. The HTML in those is correct, and test code is out of scope here.Docs-only change. No
challenge.py, tests, tolerances or starters touched.