diff --git a/docs/benchmarks.md b/docs/benchmarks.md index 0e52646..be9afd0 100644 --- a/docs/benchmarks.md +++ b/docs/benchmarks.md @@ -125,11 +125,11 @@ The smallest measured size at which Metal beats the CPU path. The break-even is | function | decision | measured | |---|---|---| -| fft2+ifft2 | CPU only: no Metal path yet (see open-items.md if it is a work item), or the work is CPU-side by nature | | -| potential build (infinite) | CPU only: no Metal path yet (see open-items.md if it is a work item), or the work is CPU-side by nature | | +| fft2+ifft2 | CPU only: no Metal path yet (see the issues if it is a work item), or the work is CPU-side by nature | | +| potential build (infinite) | CPU only: no Metal path yet (see the issues if it is a work item), or the work is CPU-side by nature | | | multislice 10 slices (plane wave, built potential) | Metal above 128² (break-even between 64² and 128²) | 64²: cpu 0.929 ms / metal 4.99 ms, 128²: cpu 11.6 ms / metal 6.53 ms, 192²: cpu 25.9 ms / metal 7.84 ms, 256²: cpu 60.1 ms / metal 10.3 ms, 512²: cpu 299 ms / metal 10.2 ms | -| transmission functions | CPU only: no Metal path yet (see open-items.md if it is a work item), or the work is CPU-side by nature | | -| probe build | CPU only: no Metal path yet (see open-items.md if it is a work item), or the work is CPU-side by nature | | +| transmission functions | CPU only: no Metal path yet (see the issues if it is a work item), or the work is CPU-side by nature | | +| probe build | CPU only: no Metal path yet (see the issues if it is a work item), or the work is CPU-side by nature | | | STEM scan 8x8, annular | Metal above 64² (break-even between <64² and 64²) | 64²: cpu 27.9 ms / metal 9.39 ms, 128²: cpu 176 ms / metal 13.8 ms, 192²: cpu 459 ms / metal 35.3 ms, 256²: cpu 714 ms / metal 50.6 ms, 512²: cpu 3.01e+03 ms / metal 176 ms | -| frozen phonons 4 x multislice, built per configuration | CPU only: no Metal path yet (see open-items.md if it is a work item), or the work is CPU-side by nature | | +| frozen phonons 4 x multislice, built per configuration | CPU only: no Metal path yet (see the issues if it is a work item), or the work is CPU-side by nature | | | multislice 10 slices (plane wave, potential built on the GPU) | Metal only (the GPU build of the Metal row above; its CPU counterpart is potential build + multislice) | | diff --git a/docs/open-items.md b/docs/open-items.md deleted file mode 100644 index 6de36fd..0000000 --- a/docs/open-items.md +++ /dev/null @@ -1,37 +0,0 @@ -# Open items - -A plain list, most important first. Each item has an issue; the issue holds the detail. Benchmarks: `docs/benchmarks.md`; -coverage: `docs/parity.md`. - -## Slower than Python at a typical size - -- **Any-length FFT with a prime factor above 7 (Bluestein)**: lengths off vDSP that factor into 2, 3, 5 and 7 run a - mixed-radix FFT since #42 (250² 0.36 ms against NumPy 0.24 ms, 500² 1.58 ms against 1.52 ms); 506² (2 × 11 × 23) stays on - Bluestein, 1.6x slower than NumPy. Numbers are right (rtol 1e−7), only speed is open. Fix candidate: generic odd-radix - passes (#4). -- **Rows measured on a loaded machine**: `fft2+ifft2` at 192² to 512², multislice and frozen phonons at 192², the probe build - at 192² to 512² against abTEM float32. Re-measure in a quiet run before changing anything (#5). - -## Work items - -- **Detector and noise follow-ups from 0.4.0**: line profiles for segmented and - pixelated line scans, `LineScan` defaults, the "spline" method of `Images.interpolate`, tilt distributions (#9). - `integrate_disc(interpolate:)` and `Images.interpolate` (method "fft") are in; the DPC measurement of a `SegmentedDetector` - is under Polar measurements. The azimuthal average of a single pattern is in (`RealField.azimuthalAverage`), over a - pixelated scan it is not. -- **Potential API**: `CrystalPotential` (a unit potential repeated, frozen phonons per cell), its API undecided (#7). -- **Cells**: non-orthogonal and rotated cells, `periodic=False`, `orthogonalize_cell` in `AtomicStructure` (the kit keeps - its own structure type: `docs/adr/0001-standalone-structure-types.md`). Finite projection, the Kirkland, Peng and Ewald - parametrisations, Debye–Waller tables (`docs/parity.md`). -- **Metal**: the bounds hold to 100 slices and grids to 1024²; re-measure past 300 slices with the renormalisation, and add - a 1024² row to the benchmark table (#5). -- **Polar measurements**: the DPC measurement of a `SegmentedDetector` is in (`integrate`, `integrateRadial`, - `differentials`); `PolarMeasurements.to_diffraction_patterns` and `azimuthal_limits` remain (#9). - -## Notes - -- **Lobato π**: the pins use `LobatoParameterization(piConstant: .abTEMFloat32)`; with the default `.exact` results differ - by about 2.6e−8 of the maximum. -- **`WavesDetector` pins**: abTEM 1.0.10's own `probe.scan(..., WavesDetector(gpts=...))` raises a broadcast error (the - detector does not declare its output shape); the pins use `probe.multislice` then `WavesDetector(gpts).detect`, abTEM's - per-wave code. diff --git a/docs/parity.md b/docs/parity.md index 5893415..e8144ea 100644 --- a/docs/parity.md +++ b/docs/parity.md @@ -234,4 +234,23 @@ counterparts are its `max_batch` and dask settings, which are not counted as mis - The CPU batch on up to `ExecutionOptions.maximumCPUThreads` (8 by default) threads, equal to sequential runs (exact today). - The constants of the CTF closed forms as parameters (`factor:` of `scherzerDefocus`, 1.5 by default as abTEM; `divisor:` of `pointResolution`, 6; `oversampling:` of `nyquistSampling`, 4). -- `LobatoParameterization.PiConstant`: the exact π by default, abTEM's float32 π on request. \ No newline at end of file +- `LobatoParameterization.PiConstant`: the exact π by default, abTEM's float32 π on request. +## Known limits and decisions + +Open work is in the [issues](https://github.com/mac4DSTEM/SimulatoKit/issues), including the ones closed as *not planned* +and parked for later (quiet-run benchmarks #5, real cells #6, FFT path #43). Speed against abTEM and NumPy is in +`docs/benchmarks.md`. + +- **FFT with a prime factor above 7:** lengths off vDSP that factor into 2, 3, 5 and 7 run a mixed-radix FFT; 506² + (2 × 11 × 23) stays on Bluestein, 1.6x slower than NumPy. The numbers are right (rtol 1e−7). +- **Rows measured on a loaded machine:** `fft2+ifft2` at 192² to 512², multislice and frozen phonons at 192², the probe build + at 192² to 512² against abTEM float32; re-measure in a quiet run before changing anything. +- **Metal:** the bounds hold to 100 slices and grids to 1024²; past 300 slices with the renormalisation is not measured. +- **Not ported:** `CrystalPotential` (a unit potential repeated, frozen phonons per cell; its API is undecided); non-orthogonal + and rotated cells, `periodic=False`, `orthogonalize_cell` (the kit keeps its own structure type, + `docs/adr/0001-standalone-structure-types.md`); finite projection; the Kirkland, Peng and Ewald parametrisations; + Debye–Waller tables; `PolarMeasurements.to_diffraction_patterns` and `azimuthal_limits`. +- **Lobato π:** the pins use `LobatoParameterization(piConstant: .abTEMFloat32)`; with the default `.exact` results differ by + about 2.6e−8 of the maximum. +- **`WavesDetector` pins:** abTEM 1.0.10's own `probe.scan(..., WavesDetector(gpts=...))` raises a broadcast error; the pins + use `probe.multislice` then `WavesDetector(gpts).detect`, abTEM's per-wave code. diff --git a/docs/status.md b/docs/status.md index c790378..c68beae 100644 --- a/docs/status.md +++ b/docs/status.md @@ -4,6 +4,6 @@ SimulatoKit 0.4.0 (2026-10-22), against abTEM 1.0.10. Live: crystal cells and th the length, a mixed-radix FFT for most other lengths, Bluestein where a large prime factor makes it the cheaper), multislice on `.cpu` (float64) and `.metal` (float32, held to the CPU), beam tilt, the probe with polar aberrations C10 to C56, `GridScan`, frozen phonons, annular, pixelated, segmented and centre-of-mass detectors, `STEM.scan`, the contrast transfer function, source size and Poisson noise, `LineScan` and line profiles, integrated centre of mass (`integrate_gradient`, iCOM), and `WavesDetector` (the exit wave at every scan position). What is checked against what: `docs/parity.md` (37 present, 36 partial, 55 missing of 128 rows); timings against Python: -`docs/benchmarks.md`; what is open: `docs/open-items.md`. +`docs/benchmarks.md`; what is open: the GitHub issues; known limits: `docs/parity.md`. Tests are split in two. `SimulatoKitParityTests` holds the pinned abTEM comparisons, the truth tests, Metal against CPU and the benchmarks (111 tests; fixtures live in its `Fixtures/`); the other targets hold the fast unit tests (35 tests); `SimulatoKitTestSupport` holds the helpers both use. `swift test -c release --parallel --skip SimulatoKitParityTests` runs the fast half; the plain `swift test -c release --parallel` runs all 146. diff --git a/tools/benchmarks/merge.py b/tools/benchmarks/merge.py index 562d854..69e7c5c 100644 --- a/tools/benchmarks/merge.py +++ b/tools/benchmarks/merge.py @@ -93,7 +93,7 @@ def sh(*cmd): out.append(f"| {f} | {why} | |") continue if not both: - why = "CPU only: no Metal path yet (see open-items.md if it is a work item), or the work is CPU-side by nature" + why = "CPU only: no Metal path yet (see the issues if it is a work item), or the work is CPU-side by nature" out.append(f"| {f} | {why} | |") continue wins = [s for s in sorted(both) if both[s]["metal"] < both[s]["cpu"] * 0.9]