Skip to content

Fix example rounding in 2 challenges and align example tests with Example 1 in 4 - #389

Open
kunal-mansukhani wants to merge 1 commit into
mainfrom
fix/example-tests-rounding-nov19
Open

kunal-mansukhani wants to merge 1 commit into
mainfrom
fix/example-tests-rounding-nov19

Conversation

@kunal-mansukhani

Copy link
Copy Markdown
Contributor

Follow-up to the example-output audit in #388. This PR is separate from #388 and branched from main. It makes two kinds of change: it fixes two rounding slips in challenge.html, and it makes generate_example_test() use the same inputs as Example 1 in four challenges. Nothing changes in reference_impl, tolerances, functional or performance tests, or starters.

1. Rounding fixes in challenge.html

challenge before after reference_impl (CPU fp32)
5_softmax, Example 1 [0.090, 0.244, 0.665] [0.090, 0.245, 0.665] 0.090031, 0.244728, 0.665241
40_batch_normalization, Example 1 [[-1.224, -1.224], [0.0, 0.0], [1.224, 1.224]] [[-1.225, -1.225], [0.0, 0.0], [1.225, 1.225]] ±1.224743

Both examples keep their 3-decimal precision. The old values were truncated instead of rounded.

2. generate_example_test() now matches Example 1

In these four challenges the HTML was already correct, but the example test used different inputs. The test now uses the HTML inputs.

challenge before after reference output, matches HTML Example 1
52_silu N=4, input [1, 2, 3, 4] N=3, input [0.5, 1.0, -0.5] [0.311230, 0.731059, -0.188770]
14_multi_agent_sim agents [0,0,1,0, 5,0,0,1] agents [0,0,1,0, 3,4,0,-1] [1,0,1,0, 3,3,0,-1]
61_rope_embedding random M=1024, D=64 M=2, D=4 with Q/Cos/Sin from Example 1 [[1,2,3,4],[-1,-1,1,1]]
64_weight_dequantization random 256x256, TILE_SIZE=128 4x4, TILE_SIZE=2 with X/S from Example 1 [[5,5,10,10],[5,5,10,10],[8,8,2,2],[8,8,2,2]]

The Example 1 Cos/Sin tables in 61 already have equal halves, which is the _half_split layout, so they're used as-is. _half_split is still used by the functional tests.

Verification

  • I ran each updated generate_example_test() through reference_impl on CPU (device="cpu") and compared the result with the HTML Example 1 output at the precision it's printed. All six challenges match. Every other example in those challenges still matches too.
  • black --check, isort --check-only and flake8 pass locally on the changed files.

- 5_softmax / 40_batch_normalization: round shown outputs correctly (0.245, ±1.225).
- 52_silu, 14_multi_agent_sim, 61_rope_embedding, 64_weight_dequantization:
  generate_example_test() now uses the Example 1 inputs from challenge.html.

@github-actions github-actions Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

⚠️ Review bot: needs a human reviewer.

  • edits existing challenge grader/tests challenges/easy/52_silu/challenge.py
  • edits existing challenge grader/tests challenges/hard/14_multi_agent_sim/challenge.py
  • edits existing challenge grader/tests challenges/medium/61_rope_embedding/challenge.py
  • edits existing challenge grader/tests challenges/medium/64_weight_dequantization/challenge.py

@github-actions github-actions Bot added the needs-human-review Review bot: needs a human reviewer label Nov 19, 2026

@shxjames shxjames left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Reviewed c4d9acd. Both rounding fixes and all four example-test alignments check out. The diff is limited to those challenge.html lines and the four generate_example_test() functions; reference_impl, tolerances, functional/perf tests, and starters are unchanged.

Rounding (HTML only). Independent fp64 matches reference_impl, and 3-decimal rounding is now consistent:

  • 5_softmax Example 1: 0.090031, 0.244728, 0.665241 → [0.090, 0.245, 0.665]. The old 0.244 was truncated. Example 2 is unchanged and already correctly rounded.
  • 40_batch_normalization Example 1: ±1.224743 → ±1.225. The old ±1.224 was truncated. Example 2 already matches.

Example-test alignment. Each generate_example_test() now uses the HTML Example 1 inputs (and shapes/params). Reference and fp64 outputs match the published HTML, and every example dict still includes the output buffer and all signature params:

  • 52_silu: N=3, [0.5, 1.0, -0.5] → [0.311230, 0.731059, -0.188770]. The HTML prints 0.3112295 / -0.1887705 (pre-existing mixed precision; within 5e-7 of fp64).
  • 14_multi_agent_sim: agents [0,0,1,0, 3,4,0,-1] → [1,0,1,0, 3,3,0,-1]. Distance is exactly 5.0, so the strict < r rule means no neighbors and the update is just position += velocity, matching the HTML.
  • 61_rope_embedding: M=2, D=4 with the Example 1 Q/Cos/Sin (already half-split) → [[1,2,3,4],[-1,-1,1,1]].
  • 64_weight_dequantization: 4×4, TILE_SIZE=2 → the exact integer HTML output.

Pre-existing, non-blocking: 64's constraints say TILE_SIZE ∈ {16,32,64,128}, but Example 1 (and now the example test) uses 2. Functional tests still use the allowed sizes.

CI is green. I have not approved.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

needs-human-review Review bot: needs a human reviewer

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants