From 83db66307712b3540202e36f424a286ef5ae3f8f Mon Sep 17 00:00:00 2001 From: Claude Date: Wed, 28 Oct 2026 17:16:15 +0000 Subject: [PATCH] docs: benchmark notes as public docs with a fresh-clone re-run docs/benchmarks.md states the machine, toolchain, method and numbers only: no development-process wording, no paths outside the kit, "not measured yet" for sizes that were not run, and a re-run command that works from a fresh clone. The numbers are unchanged. Part of mac4DSTEM/macEM-suite#81 Done: the benchmark notes read as public docs. Next: nothing in this kit for #81. Co-Authored-By: Claude Opus 5.5 Claude-Session: https://claude.ai/code/session_01VkKZESXMR4LHCwrEqzzABb --- docs/benchmarks.md | 24 +++++++++++++++++------- 1 file changed, 17 insertions(+), 7 deletions(-) diff --git a/docs/benchmarks.md b/docs/benchmarks.md index 903eaa6..6d254e6 100644 --- a/docs/benchmarks.md +++ b/docs/benchmarks.md @@ -1,12 +1,22 @@ # Benchmarks: SpectroKit against HyperSpy and exSpy -Written 2026-10-21 for 0.8.0. Harness: `tools/benchmarks/run.sh [small,typical]` (Swift: `Tests/SpectroEDXTests/BenchmarkQuantificationTests.swift`, -`Tests/SpectroEELSTests/BenchmarkThicknessTests.swift`, built by `swift test -c release`, enabled by `SPECTRO_BENCH_OUT`; Python: `tools/benchmarks/bench.py`), -under an 8 GB memory cap (measured Swift peak 1.3 GB at the typical size). Inputs are closed-form on both sides (formulas in the harness headers), float64. +Written 2026-10-21 for 0.8.0. Harness: Swift `Tests/SpectroKitParityTests/BenchmarkQuantificationTests.swift` and +`Tests/SpectroKitParityTests/BenchmarkThicknessTests.swift` (built by `swift test -c release`, enabled by `SPECTRO_BENCH_OUT`), Python +`tools/benchmarks/bench.py`, merged by `tools/benchmarks/table.py`. The Swift side peaks at 1.3 GB at the typical size. Inputs are closed-form on both sides (formulas in the harness headers), float64. One warm-up, then five runs, median and (min-max). Machine: Apple M5 Pro, 64 GB, macOS 27.0.1 (26A434), Swift 6.4 (swiftlang-6.4.0.34.1), Python 3.12.14, NumPy 2.5.4, HyperSpy 2.4.0, exSpy 0.3.2 -(no Numba). **Other sessions were running on the machine**, so the Python spreads are wide; read ratios within about 30 %. +(no Numba). **Other jobs were running on the machine**, so the Python spreads are wide; read ratios within about 30 %. + +Re-run from a fresh clone of the kit, on a quiet Mac, one benchmark at a time: + +```sh +python3.12 -m venv .venv-bench && .venv-bench/bin/pip install hyperspy==2.4.0 exspy==0.3.2 +OUT="$PWD/.build/benchmarks"; mkdir -p "$OUT" +SPECTRO_BENCH_OUT="$OUT" SPECTRO_BENCH_SIZES=small,typical swift test -c release --filter Benchmark # Swift side +.venv-bench/bin/python -I -B tools/benchmarks/bench.py "$OUT" small,typical # Python side +.venv-bench/bin/python -I -B tools/benchmarks/table.py "$OUT" # prints the table below +``` ## Results @@ -28,13 +38,13 @@ Machine: Apple M5 Pro, 64 GB, macOS 27.0.1 (26A434), Swift 6.4 (swiftlang-6.4.0. | t/lambda log ratio: estimated | typical | 4096 | 16.34 (16.29-17.02) | 87.30 (82.73-123.58) | 5.34 | Sizes: quantification small 8 x 8, typical 256 x 256 (3 elements), large 1024 x 1024; thickness small 8 x 8, typical 64 x 64 (2048 channels), large 256 x 256. -**Large: nightly** (not run in the session). +**Large: not measured yet** (`large` in the size list adds it). -## Backend decision per function (D16) +## Backend decision per function | function | decision | why | |---|---|---| -| quantification maps, all methods (the Python column converts to weight percent only; Swift returns both) | CPU only (parallel over pixels since 0.8.0) | 9 to 40 ms for 65 536 pixels; per pixel the work is 10 to 500 ns of arithmetic, so a Metal dispatch pays only above about 1e6 pixels; the break-even is **not measured** (large size is nightly). Not a work item until the nightly shows a slow large image | +| quantification maps, all methods (the Python column converts to weight percent only; Swift returns both) | CPU only (parallel over pixels since 0.8.0) | 9 to 40 ms for 65 536 pixels; per pixel the work is 10 to 500 ns of arithmetic, so a Metal dispatch pays only above about 1e6 pixels; the break-even is **not measured** (the large size is not measured yet). Not a work item until a large-size run shows a slow large image | | t/lambda log ratio, fixed threshold | CPU only; **slower than Python on the typical size** (0.49x), first in `docs/open-items.md` | single-threaded scalar passes against NumPy's vectorised axis sums | | t/lambda log ratio, estimated threshold | CPU only | 5x faster than Python at the typical size (Python's per-pixel Savitzky-Golay search dominates) | | EDX line fit, auto ID, EELS background fit, edge quantification | not benchmarked in 0.8.0 | need a Velox spectrum image or a planted EELS image on both sides; listed in `docs/open-items.md` |