level3 conv-family: accept seed to make ref ↔ cand verification possible - #145
Open
zhshgmail wants to merge 1 commit into
Open
level3 conv-family: accept seed to make ref ↔ cand verification possible#145zhshgmail wants to merge 1 commit into
seed to make ref ↔ cand verification possible#145zhshgmail wants to merge 1 commit into
Conversation
…ion possible
Affects level3/6_ConvStandard1d.py, 7_ConvStandard2d.py, 8_ConvStandard3d.py,
9_ConvDepthwise2d.py, 10_ConvTranspose2d.py.
## Problem
All five conv ops have `Model.forward()` that constructs `nn.ConvNd(...)` inline
on every call. The verification harness `utils/verification_ascendc.py:306` does
`torch.manual_seed(0)` ONCE before the per-case for-loop. Inside each iteration,
it calls `ref_model.forward(*args)` then `cand_model.forward(*args)` back-to-back.
Because `nn.ConvNd.__init__` draws random weights from the global RNG, and the
RNG state advances after the ref call, **cand draws different weights than ref**.
Verification then compares two different random outputs → ref ≠ cand by
construction, regardless of whether the candidate kernel is correct.
This makes the conv family structurally unverifiable. Hits any worker porting
`nn.Conv1d/2d/3d/Depthwise/Transpose2d` to a candidate kernel: even an exact
re-implementation in PyTorch (e.g. `F.conv1d(x, conv.weight, conv.bias, ...)`)
would diverge from the ref because the ref's `conv.weight` ≠ the ones the cand
sees.
## Fix
Add a new optional `seed` kwarg (default 0) to each conv `forward()`. At the
start of `forward()`, call `torch.manual_seed(seed)` so the inline conv
construction draws weights from a fixed RNG state. Both ref and cand pass the
same seed → both construct conv with identical weights → comparison works.
Also adds `.to(x.device)` after construction so the conv's weights track the
input device (handles the existing CPU-vs-NPU device-mismatch error that the
inline-construction pattern triggers when called with NPU input).
`get_input_groups()` extracts `seed` from the JSON case attrs (defaulting to 0)
so existing JSON cases continue to work unchanged. Adding a `seed` entry to
existing cases is optional — the default of 0 is sufficient for verification.
## Verification
Tested on A5 NPU (Ascend950PR_9589, container a5ops_dev3):
- ref_preflight on op#6 ConvStandard1d: verdict=RUNNABLE (was FAILED with
"Expected all tensors to be on the same device" before this fix)
- Two consecutive `Model().forward(x, ..., seed=42)` calls produce bit-exact
identical output for op#6/7/9/10. Verified by `torch.equal(out1, out2)`.
(op#8 ConvStandard3d skipped in test due to different x rank, but
fix is identical pattern.)
## Backward compat
- `seed` is optional with default 0. Existing JSON cases that don't include
`{"name": "seed", ...}` continue to work — `attr_inputs.get("seed", 0)`
returns 0.
- ModelNew / candidate implementations that don't pass seed will get seed=0
by default. They MUST also seed before any RNG-dependent construction
(or extract weights from the seed-0 conv) to match ref.
## Why now
Caught during op-gen cold-start of op#6 ConvStandard1d on Ascend950PR
(2026-04-28). Worker repeatedly hit the structural unverifiability before
the harness behavior was understood; downstream fix in worker model.py
(workspace-local OL-89 prose-spec extension with `torch.manual_seed`)
unblocks individual workers but the upstream fix unblocks every future
cold-start at no cost.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Problem
All five level-3 conv ops (
6_ConvStandard1d,7_ConvStandard2d,8_ConvStandard3d,9_ConvDepthwise2d,10_ConvTranspose2d) haveModel.forward()that constructsnn.ConvNd(...)inline on every call. The verification harnessutils/verification_ascendc.py:306doestorch.manual_seed(0)once before the per-case for-loop. Inside each iteration it callsref_model.forward(*args)thencand_model.forward(*args)back-to-back.Because
nn.ConvNd.__init__draws random weights from the global RNG, and the RNG state advances after the ref call, cand draws different weights than ref. Verification then compares two different random outputs → ref ≠ cand by construction, regardless of whether the candidate kernel is correct.This makes the conv family structurally unverifiable. It hits any worker porting
nn.Conv*dto a candidate kernel: even an exact PyTorch re-implementation (e.g.F.conv1d(x, conv.weight, conv.bias, ...)) would diverge from the ref because the ref'sconv.weight≠ the ones the cand sees.Discovered during an op-gen cold-start of op#6 ConvStandard1d on Ascend950PR (2026-04-28). The downstream workaround would be a workspace-local
model.pywithtorch.manual_seed(...)inside forward — but that has to be reapplied per-op per-cold-start. Fixing it upstream unblocks every future cold-start at no cost.Fix
Add a new optional
seedkwarg (default0) to each convforward(). At the start offorward(), calltorch.manual_seed(seed)so the inline conv construction draws weights from a fixed RNG state. Both ref and cand pass the same seed → both construct conv with identical weights → comparison is meaningful.Also adds
.to(x.device)after construction so the conv's weights track the input device (this is the existing CPU-vs-NPU device-mismatch error that the inline-construction pattern produces when called with NPU input — was already a separate bug, fixed here in passing).get_input_groups()readsseedfrom the JSON case attrs (defaulting to 0) so existing JSON cases continue to work unchanged.Backward compat
seedis optional with default0. Existing JSON cases that don't include{"name": "seed", ...}continue to work viaattr_inputs.get("seed", 0).seedwill getseed=0by default. They MUST also seed before any RNG-dependent construction (or extract weights from the seed-0 conv) to match ref.Verification
Tested on A5 NPU (Ascend950PR_9589, container
a5ops_dev3):(op#8 ConvStandard3d skipped in determinism test due to different x rank, but fix is identical pattern across all five files.)
Files changed
benchmarks/NPUKernelBench/level3/6_ConvStandard1d.pybenchmarks/NPUKernelBench/level3/7_ConvStandard2d.pybenchmarks/NPUKernelBench/level3/8_ConvStandard3d.pybenchmarks/NPUKernelBench/level3/9_ConvDepthwise2d.pybenchmarks/NPUKernelBench/level3/10_ConvTranspose2d.py🤖 Generated with Claude Code