Skip to content

fix(BACKEND-TENSTORRENT): route the exact-f32 decode dots through mean·K — the tt-metal pin moved f32 ttnn::sum onto the tf32 FPU tile-reduce - #3481

Merged
lu-zero merged 1 commit into
localai-org:mainfrom
lu-zero:row/TT-BACKEND-SUITE-REPAIR
Oct 19, 2026
Merged

lu-zero merged 1 commit into
localai-org:mainfrom
lu-zero:row/TT-BACKEND-SUITE-REPAIR

Conversation

@lu-zero

@lu-zero lu-zero commented Oct 19, 2026

Copy link
Copy Markdown
Collaborator

Two of the three P150 device-test failures flagged during the #3475 review share one root cause and are fixed here; the third is the pre-recorded in-suite-only RAC flake and is left to its owed bisect.

Root cause

The exact-f32 dots in the composed GDN decode step and the grouped keep-quant P==1 decode are broadcast multiply (exact f32 SFPU) + ttnn::sum over the K dim. The tt-metal stack this backend builds against since 252ca04 (tt_umd 0.9.12 era, /tmp/umdtrial-install) moved f32 ttnn::sum onto the tf32 FPU tile-reduce — the pin's own generic_reductions.hpp documents the split: mean's default fast_and_approximate_mode=false is "the accurate SFPU path (full fp32)", true is "the faster tf32 FPU path", and sum has no SFPU arm. Neither GDN nor KEEPQUANT product code changed after the 2026-10-04 102/103 suite run, so the pin advance is what broke the pins.

Red-first evidence (P150, deterministic, 3/3 identical runs each)

  • kGdnDecode wide-range arm: state max_abs 0.0051, max_rel 1.83 vs the 0.002/1e-5 envelope (test_tenstorrent_backend.cpp:4126). The error lands in the small-magnitude delta-rule terms — the exact signature 8ab35b9 pinned the arm against. VT_TT_GDN_DECODE=chunked degrades further (max_abs 0.246), ruling out a composed-only defect.
  • kMatmulBTQuantGrouped decode P=1 Q4_K N=1024 K=5120: bad=154/1024, worst_rel 0.1307 (:9696), bit-identical across the fused arm, VT_TT_KEEPQUANT_FUSED=0, and VT_TT_KEEPQUANT_MM_CHAIN=1 — the shared ttnn::sum is the defective stage, not a matmul arm.

Fix

Reduce through ttnn::mean(dim, keep_dim) and rescale by the exact f32 K (two ulp of rounding against tf32's 2^-11 truncation) at the three sites. No assertion re-pinned, no envelope widened.

Green evidence

  • GDN wide-range: state rel_rms 3.6e-5 → 3.2e-8, max_rel 1.83 → 9.1e-4.
  • Grouped decode: worst_rel 0.1307 → 1.7e-4, bad 0/1024.
  • Full suite on the P150 (2026-10-19, default order): 107/108 cases, 791117/791119 assertions. The only failure is the batched decode RAC capture-safety case (:2550/:2551), which passes isolated 3/3 and whose in-suite failure values vary run to run (126/128 subtle drift on the pre-fix baseline, 106/128 with inf after): the pre-recorded 2261-class RAC flake whose bisect is an owed item (.agents/specs/tenstorrent-decode-fusion.md:241,300), not touched here. The post-summary teardown segfault (exit 139) is the known defect recorded in 0694cdf.
  • test_kolibri1_tt host suite: 17/17 cases, 246/246 assertions.
  • agent-preflight.sh --staged, check-agent-record.py, commit style and trailer gates: all green.

Issues

  • ISSUE-LOCAL-01M590MZYN28EA6DEKFFA8GNVY (BACKEND-TENSTORRENT-GDN)
  • ISSUE-LOCAL-01M590NESYZ6D3QNF3F34DDXJG (BACKEND-TENSTORRENT-KEEPQUANT)

Filed and closed with this fix's evidence.

FOLLOWING_AGENTS_PROTOCOL

Following-Agents-Protocol: true
AI-Assisted: true
Assisted-by: AGENT:zai/glm-5.3-flash [maki]

…n·K — the tt-metal pin moved f32 ttnn::sum onto the tf32 FPU tile-reduce

Three device tests failed on the P150 during the localai-org#3475 review: the
batched decode RAC capture-safety case, the kGdnDecode wide-range arm,
and the kMatmulBTQuantGrouped P=1 decode envelope. Two of the three
share one root cause and this commit repairs those; the RAC case is the
pre-recorded in-suite-only flake documented below.

Root cause (GDN + grouped keep-quant, one defect, two sites): both
"exact f32 dot" constructions are broadcast multiply (exact f32 SFPU) +
ttnn::sum over the K dim (tenstorrent_gdn.cpp GdnDecodeStepComposed;
tenstorrent_keepquant.cpp, the fused and chunk tails of the P==1
grouped decode). The tt-metal stack this backend now builds against
(252ca04, tt_umd 0.9.12 era, /tmp/umdtrial-install built 2026-10-09)
moved f32 ttnn::sum onto the tf32 FPU tile-reduce — the pin's own
generic_reductions.hpp documents the split: mean's default
fast_and_approximate_mode=false is "the accurate SFPU path (full
fp32)", true is "the faster tf32 FPU path", and sum has no SFPU arm.
Neither GDN nor KEEPQUANT product code changed after the Oct 4
102/103 suite run, so the pin advance is what broke the pins.

Red-first evidence (P150, deterministic, 3/3 identical runs each):
- kGdnDecode wide-range: state max_abs 0.0051, max_rel 1.83 vs the
  0.002/1e-5 envelope (test_tenstorrent_backend.cpp:4126). The error
  lands in the small-magnitude delta-rule terms — the exact signature
  8ab35b9 pinned the arm against. VT_TT_GDN_DECODE=chunked degrades
  further (max_abs 0.246), ruling out a composed-only defect.
- kMatmulBTQuantGrouped decode P=1 Q4_K N=1024 K=5120: bad=154/1024,
  worst_rel 0.1307 (:9696), BIT-IDENTICAL across the fused arm,
  VT_TT_KEEPQUANT_FUSED=0, and VT_TT_KEEPQUANT_MM_CHAIN=1 — the shared
  sum is the defective stage, not a matmul arm.

Fix: reduce through ttnn::mean(dim, keep_dim) and rescale by the exact
f32 K (two ulp of rounding against tf32's 2^-11 truncation) at the
three sites. No assertion re-pinned, no envelope widened. Green:
GDN state rel_rms 3.6e-5 -> 3.2e-8, max_rel 1.83 -> 9.1e-4; grouped
decode worst_rel 0.1307 -> 1.7e-4, bad 0/1024.

Full suite on the P150 (2026-10-19, default order): 107/108 cases,
791117/791119 assertions — the only failure is the batched decode RAC
capture-safety case (:2550/:2551), which passes isolated 3/3 and whose
in-suite failure values vary run to run (126/128 subtle drift on the
pre-fix baseline, 106/128 with inf on this run): the pre-recorded
2261-class RAC flake whose bisect is an owed item
(.agents/specs/tenstorrent-decode-fusion.md:241,300), not touched
here. The post-summary teardown segfault (exit 139) is the known
defect recorded in 0694cdf.

test_kolibri1_tt host suite: 17/17 cases, 246/246 assertions.

Issues: ISSUE-LOCAL-01M590MZYN28EA6DEKFFA8GNVY (GDN),
ISSUE-LOCAL-01M590NESYZ6D3QNF3F34DDXJG (KEEPQUANT), filed and closed
with this fix.

FOLLOWING_AGENTS_PROTOCOL

Following-Agents-Protocol: true
AI-Assisted: true
Assisted-by: AGENT:zai/glm-5.3-flash [maki]
@lu-zero
lu-zero merged commit 4b3a009 into localai-org:main Oct 19, 2026
23 of 28 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant