Repository navigation
docs(record): tt-metal upstream probe — SFPI 7.88 ICE, rms_norm bias persists - #3484
Merged
Merged
Conversation
…r, rms_norm bias persists and regressed on the worst row (files ISSUE-LOCAL-01M5B60NEFSEHYZXMR6FSS1RAF) FOLLOWING_AGENTS_PROTOCOL Following-Agents-Protocol: true AI-Assisted: true Assisted-by: AGENT:zai/glm-5.3-flash [maki] References: ISSUE-LOCAL-01M5B60NEFSEHYZXMR6FSS1RAF
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Record-only: files ISSUE-LOCAL-01M5B60NEFSEHYZXMR6FSS1RAF and extends the rms_norm issue with the upstream-probe numbers.
The developer asked whether tt-metal upstream improved the rms_norm behavior behind the P150 hard flips. Answer, from building origin/main (2500c1d8f63, 1793 commits past the pin) beside the pinned evidence substrate and replaying the captured in-model rows: NO — the bias persists and regressed on the largest row (tt/host 1.0072 -> 0.9809, max_abs 0.4375, all 2560 elements differ). Upstream's rsqrt rework (LEGACY_RSQRT -> ckl::RsqrtApprox::Exact in an eltwise_chain) did not fix the input-dependence. The probe also found a blocker for any future pin advance: the official SFPI 7.88.0 tarball ICEs (rvtt_live, verify_ssa) compiling the Blackhole layernorm specialization; SFPI 7.83.0 compiles it. Patch audit: two of our local patches merged upstream; four still needed; the layernorm program-cache-hit override remains absent upstream.
Verdict recorded in the issue: do not advance the pin; re-check when SFPI >= 7.89 or an upstream rms reduce/round rework lands — the Stage-2 micro replay alone is the cheap re-check. The pinned evidence substrate was never touched. The P150 gateability decision therefore stands where PR #3483 left it: not gateable under the ratified band, residual owned by this rms issue.
FOLLOWING_AGENTS_PROTOCOL
Following-Agents-Protocol: true
AI-Assisted: true
Assisted-by: AGENT:zai/glm-5.3-flash [maki]