Skip to content

feat: allow targeted strategy smoke checks#114

Open
kaozyn wants to merge 1 commit into
zeokin:mainfrom
kaozyn:kaozyn/feat-smoke-select-transforms
Open

feat: allow targeted strategy smoke checks#114
kaozyn wants to merge 1 commit into
zeokin:mainfrom
kaozyn:kaozyn/feat-smoke-select-transforms

Conversation

@kaozyn

@kaozyn kaozyn commented Jul 7, 2026

Copy link
Copy Markdown

PR kind

  • fix
  • feat

Summary

  • add optional transform-name arguments to python -m strategy.smoke
  • allow contributors to run targeted smoke checks like python -m strategy.smoke rsvd
  • add CPU-safe tests for transform selection and unknown-name handling
  • use this PR to exercise the new feat lane automation end to end

Validation

  • uv run --extra test python -m pytest tests/test_smoke_cli.py -q
  • uv run --extra test python -m py_compile $(find matmul strategy eval tests examples -name '*.py')
  • uv run --extra test python -m pytest tests/ strategy/tests/ eval/tests/ -q

GPU Result (required for feat PRs only)

metric value
accuracy 0.0022
time complexity O(N^2·M), with default M=N//8 behaving near O(N^3)
latency 1093.16 ms
VRAM usage 713.08 MiB

Regime measured: N=12000, dtype=fp32, fill=full-rank, rank M=1500, device=RTX 4090

Raw scorecard (paste python -m eval … output or --json)
RESULT_JSON: {"config":{"n":12000,"pairs":3,"dtype":"fp32","rank_m":1500,"fill":"random","accuracy_floor":0.8,"vram_unit":"gib","device":"NVIDIA GeForce RTX 4090 (PyTorch/CUDA)","seed":11},"complexity":{"normal":"O(N^3)","smart":"O(N^2 * M)  (M=min(N,max(64,N/8))=1500 -> ~O(N^3) when M grows with N)"},"exact":{"latency_s":0.6397263603284955,"peak_vram_mib":1658.125},"transforms":{"rsvd":{"accuracy":0.0021774302182113368,"latency_s":1.0931598281798263,"peak_vram_mib":713.07958984375,"flop_ratio_vs_exact":1.7716262975778547,"gated":true,"improvement":false,"score":0.0}},"ranking":["rsvd"],"best":"rsvd"}

Checklist

  • CPU-safe validation passed (strategy.smoke if relevant, plus pytest tests/ strategy/tests/ eval/tests/).
  • If this is a feat PR, I ran the scorer on unseen couples — no hardcoding of seeds/matrices.
  • If this is a feat PR, accuracy and latency come from the same run at the same dtype.
  • If this is a feat PR, this is an improvement (every cost axis down, accuracy held) or I state honestly which axis it trades — see the one rule in CONTRIBUTING.md.
  • Correctness gates pass: python eval/tests/test_eval.py, python strategy/tests/test_subspace.py, python tests/test_correctness.py.
  • If this is a feat PR, I named the device and dtype so a reviewer can reproduce the numbers.

@github-actions github-actions Bot added area:strategy Smart strategies / transforms (strategy/) area:tests Correctness gates and test suites (tests/) status:needs-review Awaiting maintainer review type:feature Feature / improvement PR that must carry a GPU scorecard type:strategy New smart strategy / transform proposal status:queued-gpu Feat/strategy PR passed non-GPU gates and is queued for the next batched GPU evaluation labels Jul 7, 2026
@github-actions

github-actions Bot commented Jul 7, 2026

Copy link
Copy Markdown

Gate chain passed. This PR is queued for the next batched GPU evaluation window.

@github-actions github-actions Bot added eval:none Correct, but does not dominate exact on every cost axis (or gain too small to count) and removed status:queued-gpu Feat/strategy PR passed non-GPU gates and is queued for the next batched GPU evaluation labels Jul 7, 2026
@github-actions

github-actions Bot commented Jul 7, 2026

Copy link
Copy Markdown

GPU evaluation complete on RTX 5090 (mock).

  • Verdict: eval:none
    Reason: gain -1.9% does not clear the 2% significance floor
  • Track: full-rank / transform mock_transform
  • Accuracy: 0.94
  • Latency: 0.021 seconds
  • Peak VRAM: 1536.0 MiB
  • FLOP ratio vs exact: 2.5
  • Score: 10.2
  • Seed: 8878683

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

area:strategy Smart strategies / transforms (strategy/) area:tests Correctness gates and test suites (tests/) eval:none Correct, but does not dominate exact on every cost axis (or gain too small to count) status:needs-review Awaiting maintainer review type:feature Feature / improvement PR that must carry a GPU scorecard type:strategy New smart strategy / transform proposal

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant