Skip to content

Fix wrong example outputs in 1 challenge (Attention with Linear Biases) - #388

Open
kunal-mansukhani wants to merge 1 commit into
mainfrom
fix/example-outputs-audit-nov19
Open

kunal-mansukhani wants to merge 1 commit into
mainfrom
fix/example-outputs-audit-nov19

Conversation

@kunal-mansukhani

Copy link
Copy Markdown
Contributor

Audit of every challenge on main (147) for wrong example outputs in challenge.html, the same kind of bug #331 fixed in Causal Self-Attention. One real mismatch turned up, and this PR fixes it.

Fix: medium/55_attn_w_linear_bias, Example 1

output row 0 col 0 col 1 col 2 col 3
before 3.05 4.05 6.05 7.05
after 3.05 4.05 5.05 6.05
reference_impl (CPU, fp32) 3.046851 4.046851 5.046851 6.046850

Row 1 (3.93 4.93 5.93 6.93) and Example 2 (3.95 4.95; reference gives 3.953587, 4.953587) were already correct.

By hand: for query 0, the logits are [1,0,0]/sqrt(4) = [0.5,0,0] and the ALiBi bias is 0.5*(0-j) = [0,-0.5,-1]. Softmax of [0.5,-0.5,-1] is [0.6285, 0.2312, 0.1402]. Applied to V's third column [3,7,11] that gives 5.047, so every column is V's row-0 value + 2.047. The published 6.05/7.05 were off by exactly 1.

How the audit was done

  • I parsed each Example block in challenge.html, rebuilt its inputs, ran reference_impl on CPU, and compared each displayed number at the precision it is printed. Example 1 was also checked against generate_example_test().
  • 206 examples across 147 challenges. The script parsed most of them. I checked the rest (LaTeX layouts, shape-only examples, per-head views) by hand or by running reference_impl on the HTML inputs.
  • Result: 202 ok, 1 wrong (this fix), and 2 cosmetic truncations left unchanged: 5_softmax shows 0.244 for 0.2447 ("approximately"), and 40_batch_normalization shows ±1.224 for ±1.2247. 20_kmeans_clustering shows no expected output, so it had nothing to compare.

Flagged but not changed: generate_example_test() doesn't mirror Example 1 in 52_silu, 14_multi_agent_sim, 61_rope_embedding and 64_weight_dequantization. The HTML in those is correct, and test code is out of scope here.

Docs-only change. No challenge.py, tests, tolerances or starters touched.

Row 0 of the Example 1 output listed 6.05, 7.05 in the last two columns;
reference_impl gives 5.046851, 6.046850, i.e. 5.05, 6.05.

@github-actions github-actions Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

✅ Review bot: low risk, safe to merge. Lint and structure checks passed; no sensitive paths or grader edits; 2 lines changed.

  • Docs-only change: one line in challenge.html, with no challenge.py, tests, tolerances or starters touched.
  • The corrected values 5.05 and 6.05 check out by hand. Softmax of [0.5,-0.5,-1] is about [0.6285,0.2312,0.1402]. Applied to V's third column [3,7,11] that gives about 5.047, matching the PR's reference_impl result of 5.046851.
  • The corrected row 0 (3.05, 4.05, 5.05, 6.05) increases by 1 per column, consistent with row 0 of V being [1,2,3,4] (inferred from the output pattern).
  • No new challenge is added, so the duplicate, ID and starter checks do not apply. The diff is complete and contains no suspicious content.

@shxjames shxjames left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Reviewed bb66508. The Example 1 fix is correct, and Example 2 was already right.

Example 1 matches both reference_impl and an independent fp64 reimplementation of the written formula, softmax(QKᵀ/√d + α(i−j)) V, with no causal mask. Row 0 is 3.04685, 4.04685, 5.04685, 6.04685, which rounds to the new 3.05, 4.05, 5.05, 6.05. The old 6.05 / 7.05 were off by exactly 1. generate_example_test() uses the same inputs (also functional test 2). Row 1 (3.93 … 6.93) was already correct.

Example 2 (3.95, 4.95) also matches; fp64 is 3.953587, 4.953587. Printed precision is consistently 2 decimal places. There are no other examples in this challenge.

Docs-only HTML change; tests, tolerances, and starters are untouched.

Pre-existing, non-blocking: the spec is not causal (the ALiBi paper usually is), and that agrees with the reference and both examples. atol = rtol = 1e-4 leaves correct fp32 ≥150× headroom under the max-scaled grader. Perf (2048×2048×1024) is about 34 MB I/O plus 17 MB for S and ~17 GFLOP, fine on a T4. The constraints line omits d = 1024.

CI is green. I have not approved.

@claude

claude Bot commented Nov 19, 2026

Copy link
Copy Markdown
Contributor

Validated bb66508.

  • Example 1 fix is correct. I recomputed softmax(QKᵀ/√d + α(i−j))·V by hand in Python, without torch, and got row 0 = 3.0469, 4.0469, 5.0469, 6.0469 and row 1 = 3.9322 … 6.9322. These round to the new HTML values. The old 6.05 / 7.05 were off by 1.
  • T4 run. A CUDA reference solution (scores → row softmax → P·V) passed the example (--action run), then all functional tests and the performance test (--action submit). The solution was not committed.
  • Checklist. The change is docs-only (one line of challenge.html). challenge.py, the starters and the tolerances are unchanged, so the remaining checklist items are not affected by this PR.

Non-blocking, pre-existing: the performance bullet lists only M and N, not d = 1,024. That could be a follow-up.

LGTM.

🤖 Generated with Claude Code

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants