Fix 14 GPU-only test failures on gfx942

[demandal25]

Summary

Fixes 14 of the 18 test failures on MI300X / gfx942. The suite was 18-failed/302-passed on GPU while CI stayed green, because every failing test is gated on requires_torch_cuda / device_count() == 0 and this fork's CI is CPU-only.

None of these failures are ROCm-specific. Both root causes are in the tests, not in production code, and would fail identically on NVIDIA.

Root cause 1 — tests skipped a config-resolution step (12 failures)

There are two config types: BenchmarkConfig (what you write; every eval field is Optional and defaults to None) and ResolvedEvalConfig (what evaluators consume; every field populated). cfg.resolve_eval_config(definition) converts one to the other.

The tests passed the unresolved BenchmarkConfig straight to Evaluator.evaluate(), which is annotated for ResolvedEvalConfig. Python does not enforce annotations at runtime, so it ran until the tolerances were actually used:

flashinfer_bench/bench/utils.py:113
    exceeds_tol_mask = (abs_error > cfg.atol) & (rel_error > cfg.rtol)
E   TypeError: '>' not supported between instances of 'Tensor' and 'NoneType'

Production is unaffected — isolated_runner.py:49 and persistent_runner.py:253 both resolve first. The tests now do the same, at all 25 call sites across three files.

Root cause 2 — a stale trace-path assertion (1 failure)

test_benchmark_with_mixed_results asserted traces/{op_type}/{name}.jsonl. add_traces has written traces/{author}/{op_type}/{name}.jsonl since d3f798c ("Adapt codebase to author-based trace directory layout", flashinfer-ai#379). The code is right and the test was never migrated — verified both directions: the file lands at traces/tester/op/simple_add.jsonl, and TraceSet.from_path rediscovers both traces because the loader rglobs recursively. CLAUDE.md documented the same stale layout (and hardcoded solutions/baseline/, where baseline is one author value) and is corrected to match.

Preventing recurrence

Evaluator.evaluate() now rejects a non-ResolvedEvalConfig at the boundary with a message naming the fix, rather than failing 100 lines deeper with a Tensor > NoneType error that says nothing about config resolution. This is the only production change in the PR; correct callers are unaffected.

Also ignores GPU run artifacts — each suite run on gfx942 leaves a ~900MB root-owned core from the MLA crashes plus .rocprofv3/, which make git add -A fail outright.

Not fixed here

• The 4 test_mla_paged failures remain red. They are an amd-flashinfer coverage gap (module 'flashinfer' has no attribute 'mla'), not something this PR can address, and are deliberately left failing rather than skipped so the gap stays visible. This PR does not make the GPU suite green.
• A tests/serve ordering flake present in the original run has not recurred in three full runs since. The 12 evaluator tests previously errored mid-suite, which is the likely source of the state it tripped on.

Follow-ups this work surfaced (separate PRs)

• flashinfer_bench/cli/main.py:111,155 use trace_set.workload; the field is workloads. flashinfer-bench report merge dies with AttributeError on any input.
• scripts/collect_stream.py:350,363 still use the pre-Adapt codebase to author-based trace directory layout flashinfer-ai/flashinfer-bench#379 flat trace path, so push_trace never finds the file, logs a warning, and the pipeline reports success having uploaded no traces.
• trace_set.py:711 interpolates raw solution.author into an on-disk path with no sanitization, while cli/main.py:147 guards the analogous solutions path with _safe_path_segment.
• The GPU workflow runs only tests/compile tests/agent tests/integration/test_aiter_generator.py — it excludes tests/bench, where 13 of these 18 failures lived. Worth widening once a runner is attached; note that adding tests/integration/flashinfer would go red on the MLA gap above.

Test plan

• MI300X / gfx942, ROCm 7.2, in the docker/rocm container: 18 failed / 302 passed → 4 failed / 325 passed, stable across runs (325 vs 316 earlier reflects ROCm: fix rocprofv3 region scoping (marker column) #12's new tests, merged in during this work).
• Confirmed the boundary guard raises the intended TypeError for the old call shape.
• CPU: 226 passed, unchanged.
• pre-commit run -a

[Copilot]

Pull request overview

Fixes GPU-only test failures by ensuring evaluator paths consume fully-resolved eval configs (matching production runners), updates the benchmark trace-path assertion to the author-scoped layout, and adds a defensive boundary check in Evaluator.evaluate() to fail fast when an unresolved config is passed.

Changes:

• Update bench tests to call BenchmarkConfig.resolve_eval_config(definition) (and use ResolvedEvalConfig directly where appropriate).
• Fix test_benchmark_with_mixed_results to assert the author-scoped trace layout (traces/{author}/{op_type}/{definition}.jsonl).
• Add a runtime type guard in Evaluator.evaluate() and update docs/ignore patterns for ROCm/GPU-run artifacts.

Reviewed changes

Copilot reviewed 7 out of 8 changed files in this pull request and generated no comments.

Show a summary per file

File	Description
tests/bench/test_isolated_runner.py	Use ResolvedEvalConfig when calling compute_error_stats (matches function contract).
tests/bench/test_evaluator.py	Resolve BenchmarkConfig into ResolvedEvalConfig before calling evaluator entrypoints.
tests/bench/test_dsa_topk_indexer_evaluator.py	Resolve eval config before evaluation to avoid None tolerances.
tests/bench/test_dsa_sparse_attention_evaluator.py	Resolve eval config before evaluation to avoid None tolerances.
tests/bench/test_benchmark.py	Update trace-file assertion to author-scoped trace directory layout.
flashinfer_bench/bench/evaluators/evaluator.py	Add fast-fail TypeError when cfg is not a ResolvedEvalConfig.
CLAUDE.md	Fix dataset path documentation to author-scoped solutions/ and traces/ layouts.
.gitignore	Ignore GPU/rocprof artifacts (needs scoping adjustment per review comment).

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.