Repository navigation
fix(BACKEND-TENSTORRENT): route the exact-f32 decode dots through mean·K — the tt-metal pin moved f32 ttnn::sum onto the tf32 FPU tile-reduce - #3481
Merged
Conversation
…n·K — the tt-metal pin moved f32 ttnn::sum onto the tf32 FPU tile-reduce Three device tests failed on the P150 during the localai-org#3475 review: the batched decode RAC capture-safety case, the kGdnDecode wide-range arm, and the kMatmulBTQuantGrouped P=1 decode envelope. Two of the three share one root cause and this commit repairs those; the RAC case is the pre-recorded in-suite-only flake documented below. Root cause (GDN + grouped keep-quant, one defect, two sites): both "exact f32 dot" constructions are broadcast multiply (exact f32 SFPU) + ttnn::sum over the K dim (tenstorrent_gdn.cpp GdnDecodeStepComposed; tenstorrent_keepquant.cpp, the fused and chunk tails of the P==1 grouped decode). The tt-metal stack this backend now builds against (252ca04, tt_umd 0.9.12 era, /tmp/umdtrial-install built 2026-10-09) moved f32 ttnn::sum onto the tf32 FPU tile-reduce — the pin's own generic_reductions.hpp documents the split: mean's default fast_and_approximate_mode=false is "the accurate SFPU path (full fp32)", true is "the faster tf32 FPU path", and sum has no SFPU arm. Neither GDN nor KEEPQUANT product code changed after the Oct 4 102/103 suite run, so the pin advance is what broke the pins. Red-first evidence (P150, deterministic, 3/3 identical runs each): - kGdnDecode wide-range: state max_abs 0.0051, max_rel 1.83 vs the 0.002/1e-5 envelope (test_tenstorrent_backend.cpp:4126). The error lands in the small-magnitude delta-rule terms — the exact signature 8ab35b9 pinned the arm against. VT_TT_GDN_DECODE=chunked degrades further (max_abs 0.246), ruling out a composed-only defect. - kMatmulBTQuantGrouped decode P=1 Q4_K N=1024 K=5120: bad=154/1024, worst_rel 0.1307 (:9696), BIT-IDENTICAL across the fused arm, VT_TT_KEEPQUANT_FUSED=0, and VT_TT_KEEPQUANT_MM_CHAIN=1 — the shared sum is the defective stage, not a matmul arm. Fix: reduce through ttnn::mean(dim, keep_dim) and rescale by the exact f32 K (two ulp of rounding against tf32's 2^-11 truncation) at the three sites. No assertion re-pinned, no envelope widened. Green: GDN state rel_rms 3.6e-5 -> 3.2e-8, max_rel 1.83 -> 9.1e-4; grouped decode worst_rel 0.1307 -> 1.7e-4, bad 0/1024. Full suite on the P150 (2026-10-19, default order): 107/108 cases, 791117/791119 assertions — the only failure is the batched decode RAC capture-safety case (:2550/:2551), which passes isolated 3/3 and whose in-suite failure values vary run to run (126/128 subtle drift on the pre-fix baseline, 106/128 with inf on this run): the pre-recorded 2261-class RAC flake whose bisect is an owed item (.agents/specs/tenstorrent-decode-fusion.md:241,300), not touched here. The post-summary teardown segfault (exit 139) is the known defect recorded in 0694cdf. test_kolibri1_tt host suite: 17/17 cases, 246/246 assertions. Issues: ISSUE-LOCAL-01M590MZYN28EA6DEKFFA8GNVY (GDN), ISSUE-LOCAL-01M590NESYZ6D3QNF3F34DDXJG (KEEPQUANT), filed and closed with this fix. FOLLOWING_AGENTS_PROTOCOL Following-Agents-Protocol: true AI-Assisted: true Assisted-by: AGENT:zai/glm-5.3-flash [maki]
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Two of the three P150 device-test failures flagged during the #3475 review share one root cause and are fixed here; the third is the pre-recorded in-suite-only RAC flake and is left to its owed bisect.
Root cause
The exact-f32 dots in the composed GDN decode step and the grouped keep-quant P==1 decode are broadcast multiply (exact f32 SFPU) +
ttnn::sumover the K dim. The tt-metal stack this backend builds against since 252ca04 (tt_umd 0.9.12 era,/tmp/umdtrial-install) moved f32ttnn::sumonto the tf32 FPU tile-reduce — the pin's owngeneric_reductions.hppdocuments the split:mean's defaultfast_and_approximate_mode=falseis "the accurate SFPU path (full fp32)",trueis "the faster tf32 FPU path", andsumhas no SFPU arm. Neither GDN nor KEEPQUANT product code changed after the 2026-10-04 102/103 suite run, so the pin advance is what broke the pins.Red-first evidence (P150, deterministic, 3/3 identical runs each)
kGdnDecodewide-range arm: state max_abs 0.0051, max_rel 1.83 vs the 0.002/1e-5 envelope (test_tenstorrent_backend.cpp:4126). The error lands in the small-magnitude delta-rule terms — the exact signature 8ab35b9 pinned the arm against.VT_TT_GDN_DECODE=chunkeddegrades further (max_abs 0.246), ruling out a composed-only defect.kMatmulBTQuantGroupeddecode P=1 Q4_K N=1024 K=5120: bad=154/1024, worst_rel 0.1307 (:9696), bit-identical across the fused arm,VT_TT_KEEPQUANT_FUSED=0, andVT_TT_KEEPQUANT_MM_CHAIN=1— the sharedttnn::sumis the defective stage, not a matmul arm.Fix
Reduce through
ttnn::mean(dim, keep_dim)and rescale by the exact f32 K (two ulp of rounding against tf32's 2^-11 truncation) at the three sites. No assertion re-pinned, no envelope widened.Green evidence
:2550/:2551), which passes isolated 3/3 and whose in-suite failure values vary run to run (126/128 subtle drift on the pre-fix baseline, 106/128 with inf after): the pre-recorded 2261-class RAC flake whose bisect is an owed item (.agents/specs/tenstorrent-decode-fusion.md:241,300), not touched here. The post-summary teardown segfault (exit 139) is the known defect recorded in 0694cdf.test_kolibri1_tthost suite: 17/17 cases, 246/246 assertions.agent-preflight.sh --staged,check-agent-record.py, commit style and trailer gates: all green.Issues
Filed and closed with this fix's evidence.
FOLLOWING_AGENTS_PROTOCOL
Following-Agents-Protocol: true
AI-Assisted: true
Assisted-by: AGENT:zai/glm-5.3-flash [maki]