Skip to content

Add a criterion benchmark suite so LIMITATIONS performance numbers are re-runnable here #71

Description

@vyncint

Summary

docs/LIMITATIONS.md promises "numbers come from re-runnable commands", and every number in it (7.4–7.6 s Metal vs 6.1–6.4 s CPU; "CPU reductions 4–44× faster"; "2.6× on the motivating benchmark") was produced by oxmega device-bench in another repository, which pins oxmera = "0.2". Nothing in this repository can reproduce a 0.3 or 0.4 number: there is no benches/ directory, no criterion in the lockfile and no [profile.release].

Where (main at 853a224)

  • docs/LIMITATIONS.md "Performance honesty"; ROADMAP.md 0.2.0 and 0.3.0 paragraphs; crates/oxmera-tensor/src/cpu.rs:60-62,481-484 cite "relu: 0.85 ms → 2.3 ms per 10M" from an unnamed harness.
  • find . -path ./target -prune -o -type d -name benches -print → nothing.

Why it matters

Every remaining ROADMAP item under "Next" (buffer pooling, MPS/cuBLAS, multi-stream, f16) is gated on "measured before/after numbers". Without a harness in the repository those items cannot start, and the honesty rule cannot be checked by anyone but the maintainer.

Fix

A benches/ crate (criterion, harness = false) covering unary, binary, reductions, matmul and the Adam step at three sizes per available device, selecting devices at run time and skipping absent ones; a just bench recipe; a docs/perf/ directory holding dated result files with the machine and commit named. Add [profile.release] (lto = "thin", codegen-units = 1) or document why the default profile is the one measured.

Done when

just bench runs on a CPU-only box and on macOS; the LIMITATIONS numbers are regenerated from it and cite the file under docs/perf/.

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    enhancementNew feature or request

    Projects

    No projects

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions