Feat/roofline guided routing - #385
Open
zihaoanllm wants to merge 6 commits into
Open
Conversation
Adds knowledge/analysis_skills/ — a pluggable slot for profile-analysis skills — and
its first skill, `roofline`, which estimates how much of the hardware ceiling each
Top-N kernel already reaches and therefore how much is actually left to win.
Markdown-first per the skill contract: the logic lives in SKILL.md so an agent can
execute it by reading, and roofline_tools.py holds only mechanical primitives (peak
parsing, counter parsing, unit math). A broken helper cannot disable the skill and a
broken skill cannot fail the run — five explicit degradation levels, ending in
"emit nothing, caller behaves exactly as before".
Doctrine: roofline_pct measures how well a kernel executes its current byte/FLOP
budget, not whether that budget is necessary. A saturated kernel is done with
micro-tuning, NOT done being optimized — it routes to a byte-reduction track rather
than being dropped, which matters most precisely for the largest kernel. Levers must
preserve the measurement contract: the user-supplied workload (isl/osl/conc/batch) is
fixed and speculative decoding is not an optimization.
Calibrated against a real run (Qwen3.5-35B-A3B-FP8, gfx950, vLLM TP1, isl/osl 1k,
conc 64), reproduced by `roofline_tools.py --selftest`:
fused_moe 26.45% GPU, 88% of roofline -> 1.02x attainable, +0.58% predicted
(measured: 1.047x isolated, -0.064% e2e)
paged_attn 8.86% GPU, 18% of roofline -> 2.80x attainable, +5.70% predicted
(measured: 1.56x isolated)
Ranking by pct_gpu_time puts MoE first; ranking by roofline headroom puts attention
first — which is the entire point of the skill.
Co-Authored-By: Claude <noreply@anthropic.com>
Wires the roofline analysis skill into Profile -> Strategize -> Milestone. Orchestration: ANALYSIS_SKILL_INPUTS mirrors the existing TRACELENS_INPUTS idiom — an optional prior whose inputs are '' when off, injected at the 4 full-profile Profiler calls, the 3 Architect routing calls, and the experience/report calls. `analysis_skill=none` yields empty inputs and pre-feature behavior. Profiler: after the Top-N is final, execute ANALYSIS_SKILL_DIR/SKILL.md and emit profile_roofline.json alongside it. It enriches — it never edits profile_topN.json and never changes a pct_gpu_time. A skill that errors is noted and the Top-N is returned anyway. Architect: new step 1c consumes the prior to answer what pct_gpu_time cannot — is this kernel's time actually recoverable? Ranked candidates now carry roofline_pct, attainable_speedup, expected_e2e_gain_pct, headroom_class and confidence, and both orderings (by %GPU and by expected gain) are reported side by side so a disagreement is visible rather than blended away. The load-bearing constraint is that a saturated head is REROUTED, not dropped: new step 1d enumerates byte-reduction levers (fuse an adjacent op away, stop reading unrouted experts, layout/packing, lower-precision weights) for a kernel that is at the wall but still the biggest consumer. Those levers may not change the measurement contract — the user-supplied isl/osl/conc/batch is fixed and speculative decoding is not an optimization. Guardrails throughout: confidence=low is display-only, roofline may never prune a candidate, and a measured speedup exceeding the prediction means the model was wrong (prefer the measurement). update_experience records predicted-vs-actual into backend_playbook.md so the target_eff priors self-correct across runs. Co-Authored-By: Claude <noreply@anthropic.com>
test_roofline_skill.py (20 tests, stdlib unittest, no GPU) pins the skill against a
real run whose outcome is known — Qwen3.5-35B-A3B-FP8 on gfx950/vLLM/TP1/1k/conc64:
* fused_moe 26.45% GPU -> ~88% of roofline, saturated, ~1.02x attainable
(measured 1.047x isolated, -0.064% e2e: budget wasted)
* paged_attn 8.86% GPU -> ~18% of roofline, underperforming, >1.4x attainable
(measured 1.56x isolated)
The load-bearing case is test_rankings_disagree: %GPU ranks MoE first, headroom ranks
attention first, and reality agreed with headroom. Also pinned:
test_88pct_against_a_90pct_target_is_saturated_not_moderate, a regression guard for
banding on target_eff rather than the raw roofline — classifying 0.88-vs-0.90 as
"moderate" would route that head straight back to the wasted-budget path.
Degradation is asserted to be non-fatal at every level: unknown gfx, unusable input,
unknown dtype, impossible result (clamped + flagged suspect, raw preserved), missing
counters (None = "not measured", never 0.0 = "measured nothing"). A doc-consistency
test fails if the SKILL.md target_eff table drifts from TARGET_EFF in the helper.
test_analysis_skill_off_identical.js mirrors the existing expert-skills guard: OFF by
every spelling ('none'/'false'/''/whitespace) yields empty inputs, the object shape is
identical on and off so a spread can never add or drop a key, and the ANALYSIS_SKILL_*
keys appear as object keys nowhere outside the gating block, so the spread cannot
shadow another input.
Co-Authored-By: Claude <noreply@anthropic.com>
Adds the analysis_skill arg to the invocation block and a Roofline section covering the doctrine (advisory, never prunes; saturated means rerouted to byte-reduction, not dropped), the staged confidence model, the degradation ladder, the workload-contract constraint, and the real-run calibration table showing why %GPU alone misroutes. Co-Authored-By: Claude <noreply@anthropic.com>
…failed Three defects surfaced by the first live run (Qwen3.5-35B-A3B-FP8, gfx950). 1. A clamped 100% was being read as a verdict. fused_moe came back roofline_pct=1.00 + suspect and classified `saturated` — but the clamp came from an over-counting byte model, not from the kernel being at the wall. An infeasible ratio is evidence the model is wrong, so it must produce NO verdict: headroom_class="unknown", no attainable_speedup, plus bytes_upper_bound (= peak_bw x t, what the model violated) to aim a stage-C measurement. This was also an inconsistency with SKILL.md L3, which already said "display-only". 2. An invented bound_type. The run emitted "dispatch", a category the consumer had no routing rule for. Dispatch-bound is real, so it becomes a first-class outcome: a launch within launch-overhead scale is timed by dispatch rather than by its transfer or math, so it gets bound_type="latency", no verdict, and a note that the lever is fusion/graph capture. bound_type is now a closed set and anything else degrades to "unknown". 3. Sub-threshold kernels were modelled and mostly failed — 12 "degraded" entries out of 25, drowning the 5 that mattered. Analysis is now head-scoped (pct_gpu_time >= HEAD_THRESHOLD_PCT, biggest first, capped): below the bar the Amdahl ceiling is under the noise band anyway, so a headroom estimate cannot change a decision. Skipped entries are ABSENT, not "degraded" — a kernel too small to matter is not a modelling failure and must not read as one. Also adds a general feasibility rule to SKILL.md: with several candidate byte models, take the largest that satisfies bytes <= peak_bw * t; if none is feasible, emit no verdict. Generic — no per-op or per-model special-casing. Tests 20 -> 26. Calibration unchanged (MoE 88% saturated, attention 18% underperforming, rankings still invert). Co-Authored-By: Claude <noreply@anthropic.com>
…ing lessons in roofline skill: - bound_type is now decided by measured utilization on BOTH axes, not AI-vs-ridge alone. A small AI with low HBM util is latency/occupancy-bound, not memory-bound (the most common mislabel). Latency-bound kernels above the dispatch floor KEEP their headroom verdict (so a low-util head like paged attention still ranks by headroom) but route to occupancy/fusion levers, not byte reduction. - prefer the memory axis; compute-axis roofline_pct is untrustworthy without a validated peak (BF16 MFMA microbench reads ~2x low -> >100% artifact, tripped by L3). - routing: a head kernel with no editable call site (asm .co / precompiled .so) -> host-side levers only, regardless of headroom. - doctrine: isolated-kernel noise band (~3.4%) is much wider than the e2e serving band (~0.5%); judge isolated wins against the wider one. - tests 26 -> 29 (dual-axis bound_type; latency-with-verdict vs dispatch-no-verdict). kernel_workflow profiling guide: - fix the dep-wait -> memory-bound conflation: dependency-wait and issue-wait are the two latency sub-cases (C1 shorten chain / C2 raise occupancy) with OPPOSITE fixes; the split is a property of the tile config, re-read after every tile change. - how to read the Wavefront table (independent accumulators over Wave Cycles, never subtract; averages hide wave divergence). - cheap pre-hypothesis checks: validate peaks (>100% = mis-calibrated), GPU fill (CTAs < CU count), spill/LDS/coalescing. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Roofline-guided kernel routing
pct_gpu_time shows where time is spent but not whether it's recoverable — a kernel already at its hardware ceiling can't benefit from optimization budget.
This PR adds roofline analysis so routing can account for headroom, not just cost.
Key components
percentage, attainable speedup, and expected end-to-end gain. The Architect reports two orderings (by %GPU and by expected gain) rather than silently
combining them.
no recoverable gain), while attention used 8.86% GPU at only 18% of roofline (a real ~1.56× win). Ranking by %GPU misdirects budget to MoE; ranking by
headroom correctly targets attention.
lower precision). The measurement contract is fixed — user-supplied isl/osl/conc/batch stays constant, and speculative decoding doesn't count as optimization.
in "emit nothing, behave as before." With analysis_skill=none, behavior is byte-identical to before the feature. Analysis is head-scoped; a dispatch-bound
launch and an infeasible byte model both yield no verdict rather than a false "saturated."
test_analysis_skill_off_identical.js verifies the OFF path injects nothing.
End-to-end verification
validated end-to-end on the calibration config above (Qwen3.5-35B-A3B-FP8 / gfx950 / vLLM TP1 / 1k-1k / conc 64). The skill ran every round, degraded non-fatally, and the predicted-vs-actual calibration confirmed the routing behaves as designed. No issues.