Summary
docs/LIMITATIONS.md promises "numbers come from re-runnable commands", and every number in it (7.4–7.6 s Metal vs 6.1–6.4 s CPU; "CPU reductions 4–44× faster"; "2.6× on the motivating benchmark") was produced by oxmega device-bench in another repository, which pins oxmera = "0.2". Nothing in this repository can reproduce a 0.3 or 0.4 number: there is no benches/ directory, no criterion in the lockfile and no [profile.release].
Where (main at 853a224)
docs/LIMITATIONS.md "Performance honesty"; ROADMAP.md 0.2.0 and 0.3.0 paragraphs; crates/oxmera-tensor/src/cpu.rs:60-62,481-484 cite "relu: 0.85 ms → 2.3 ms per 10M" from an unnamed harness.
find . -path ./target -prune -o -type d -name benches -print → nothing.
Why it matters
Every remaining ROADMAP item under "Next" (buffer pooling, MPS/cuBLAS, multi-stream, f16) is gated on "measured before/after numbers". Without a harness in the repository those items cannot start, and the honesty rule cannot be checked by anyone but the maintainer.
Fix
A benches/ crate (criterion, harness = false) covering unary, binary, reductions, matmul and the Adam step at three sizes per available device, selecting devices at run time and skipping absent ones; a just bench recipe; a docs/perf/ directory holding dated result files with the machine and commit named. Add [profile.release] (lto = "thin", codegen-units = 1) or document why the default profile is the one measured.
Done when
just bench runs on a CPU-only box and on macOS; the LIMITATIONS numbers are regenerated from it and cite the file under docs/perf/.
Summary
docs/LIMITATIONS.mdpromises "numbers come from re-runnable commands", and every number in it (7.4–7.6 s Metal vs 6.1–6.4 s CPU; "CPU reductions 4–44× faster"; "2.6× on the motivating benchmark") was produced byoxmega device-benchin another repository, which pinsoxmera = "0.2". Nothing in this repository can reproduce a 0.3 or 0.4 number: there is nobenches/directory, no criterion in the lockfile and no[profile.release].Where (main at 853a224)
docs/LIMITATIONS.md"Performance honesty";ROADMAP.md0.2.0 and 0.3.0 paragraphs;crates/oxmera-tensor/src/cpu.rs:60-62,481-484cite "relu: 0.85 ms → 2.3 ms per 10M" from an unnamed harness.find . -path ./target -prune -o -type d -name benches -print→ nothing.Why it matters
Every remaining ROADMAP item under "Next" (buffer pooling, MPS/cuBLAS, multi-stream,
f16) is gated on "measured before/after numbers". Without a harness in the repository those items cannot start, and the honesty rule cannot be checked by anyone but the maintainer.Fix
A
benches/crate (criterion,harness = false) covering unary, binary, reductions, matmul and the Adam step at three sizes per available device, selecting devices at run time and skipping absent ones; ajust benchrecipe; adocs/perf/directory holding dated result files with the machine and commit named. Add[profile.release](lto = "thin",codegen-units = 1) or document why the default profile is the one measured.Done when
just benchruns on a CPU-only box and on macOS; the LIMITATIONS numbers are regenerated from it and cite the file underdocs/perf/.