fix(matmul): budget fp16 fp32-upcast temporaries in _fits_in_core (#63)

[philluiz2323]

Fixes #63.

Summary

_fits_in_core decides in-core vs tiled execution by estimating peak device residency as 3 * n * n * cfg.item_bytes (A + B + C). For the default fp16 + accumulate_fp32=True path that under-budgets by ~2.7x, because _gemm_in_core does not just hold three fp16 buffers — during the matmul it keeps, all live at once:

tensor	bytes
a (fp16)	2·n²
b (fp16)	2·n²
a.astype(fp32)	4·n²
b.astype(fp32)	4·n²
fp32 product	4·n²
total	16·n²

So the true peak is 16·n² but the check budgets 6·n² (16/6 ≈ 2.67x). A run can therefore pass the in-core check and then OOM mid-multiply instead of falling back to the tiled path — the opposite of the safe behaviour the dispatcher exists to guarantee.

The tiled path already accounts for this fp16 upcast (_tile_operand_bytes / _tile_workspace_bytes_per_elem); the in-core path was simply never given the same treatment. This is the still-open bug PR #68 targeted before it was closed for staleness — rebuilt on current main.

Fix

• Add _in_core_bytes_per_elem(cfg) — peak in-core bytes per element, mirroring _tile_operand_bytes on the tiled side:
  • fp16 + accumulate_fp32: 2*item + 3*acc = 16
  • otherwise (fp32, fp64, fp16-raw): 3*item
• Use it in _fits_in_core. fp32/fp64/fp16-raw budgets are unchanged.
• CPU-only tests: the per-element accounting per dtype, plus a fits/does-not-fit boundary where a budget that clears the old 6·n² estimate but not the true 16·n² is now correctly judged out-of-core.

Scope note

This is a VRAM-accounting correctness fix in the exact engine (matmul/gemm.py); it adds no strategy/transform and changes no scored numeric path, so there is no accuracy/latency improvement to claim. The scorecard below is the unchanged baseline on the reference regime, included per CONTRIBUTING to show the eval pipeline runs green on this branch.

Result (N=12000, full-rank, fp32, RTX 4090 (24 GB); baseline — no strategy added)

metric	value
accuracy	0.0022 (rsvd on full-rank — unchanged baseline; below the 0.8 floor, gated to 0 as usual)
time complexity	normal O(N^3), smart O(N^2·M) — unchanged
latency	1093.16 ms (rsvd) vs 639.73 ms exact — unchanged baseline
VRAM usage	713.08 MiB (rsvd) vs 1658.13 MiB exact — unchanged baseline

RESULT_JSON: {"config":{"n":12000,"pairs":3,"dtype":"fp32","rank_m":1500,"fill":"random","accuracy_floor":0.8,"vram_unit":"gib","device":"NVIDIA GeForce RTX 4090 (PyTorch/CUDA)","seed":11},"complexity":{"normal":"O(N^3)","smart":"O(N^2 * M)  (M=min(N,max(64,N/8))=1500 -> ~O(N^3) when M grows with N)"},"exact":{"latency_s":0.6397263603284955,"peak_vram_mib":1658.125},"transforms":{"rsvd":{"accuracy":0.0021774302182113368,"latency_s":1.0931598281798263,"peak_vram_mib":713.07958984375,"flop_ratio_vs_exact":1.7716262975778547,"gated":true,"improvement":false,"score":0.0}},"ranking":["rsvd"],"best":"rsvd"}

Human-readable scorecard — python -m eval --n 12000 --pairs 3 on RTX 4090, fp32

=== eval report (n=12000, 3 couples, fp32, fill=random, M=1500, seed=11) ===
  complexity : normal O(N^3)   smart O(N^2 * M)
  exact      :   639.73 ms   peak   1658.1 MiB
  transform   accuracy      latency   peakVRAM   FLOPx        score
  -----------------------------------------------------------------
  rsvd          0.0022    1093.16ms    713.1MiB    1.8x            0  (gated: low accuracy)
  best (highest score): rsvd

Reference regime is A100 (80 GB) per CONTRIBUTING; measured here on an RTX 4090 (24 GB) — device named so numbers are reproducible.

Checklist

• I ran the scorer on unseen couples — no hardcoding of seeds/matrices.
• Accuracy and latency come from the same run at the same dtype. (fp32, seed 11 above)
• This is an improvement (every cost axis down, accuracy held) or I
  state honestly which axis it trades. Correctness fix: makes the in-core VRAM estimate match true peak residency so fp16 runs stop OOMing; no cost axis changes, no accuracy traded.
• I named the device and dtype so a reviewer can reproduce the numbers. (RTX 4090, fp32)

Test plan (run on RTX 4090)

• python tests/test_auto_tile_budget.py — 6/6, incl. new test_in_core_bytes_accounts_for_fp16_upcast, test_fits_in_core_rejects_fp16_upcast_boundary, test_fits_in_core_accepts_when_true_budget_fits
• python eval/tests/test_eval.py — 16/16 pass
• python strategy/tests/test_subspace.py — 12/12 pass
• python tests/test_correctness.py — 7/8 pass. The one failure, test_fp16_incore_tiled_parity, is pre-existing and unrelated: it also fails on clean zeokin:main on the same GPU, and this PR only changes the _fits_in_core dispatch decision, not fp16 numerics.

<----ia--------e------- relabel refresh: 2026-07-07T14:23:28Z -->

[github-actions]

Please add a filled-in scorecard from python -m eval (see CONTRIBUTING.md).

[github-actions]

Please add a filled-in scorecard from python -m eval (see CONTRIBUTING.md).

[github-actions]

Gate chain passed. This PR is queued for the next batched GPU evaluation window.