Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
11 changes: 11 additions & 0 deletions .github/dependabot.yml
Original file line number Diff line number Diff line change
@@ -0,0 +1,11 @@
version: 2
updates:
# GitHub Actions only, monthly, one grouped PR; it merges itself when CI is green (dependabot-auto-merge.yml).
# Swift packages stay out: kit and app pins are exact and moved on purpose.
- package-ecosystem: github-actions
directory: /
schedule:
interval: monthly
groups:
github-actions:
patterns: ["*"]
11 changes: 11 additions & 0 deletions .github/workflows/ci.yml
Original file line number Diff line number Diff line change
Expand Up @@ -69,3 +69,14 @@ jobs:
swift --version | grep -E "Swift version 6\.([2-9]|[1-9][0-9])" || { echo "Swift 6.2 or newer needed"; exit 1; }
- name: Gate (tools/test.sh, release)
run: tools/test.sh

private-refs:
name: No private references
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v4
- name: Scan tracked files
run: |
if git grep -n -I -E '_archive|CLAUDE\.md|memcap|macEM-suite|setup-python-refs' -- ':!.gitignore' ':!.github/workflows/ci.yml'; then
echo "Private references above; this repo is public."; exit 1
fi
2 changes: 1 addition & 1 deletion .github/workflows/docs.yml
Original file line number Diff line number Diff line change
Expand Up @@ -2,7 +2,7 @@
# published to GitHub Pages at https://mac4dstem.github.io/SpectroKit/documentation/ on every tag.
# On a pull request the site is built and uploaded as an artifact only, so a broken catalog fails before the tag.
#
# No swift-docc-plugin: the kit takes no package dependencies (CLAUDE.md). `xcodebuild docbuild` builds the archives
# No swift-docc-plugin: the kit takes no package dependencies. `xcodebuild docbuild` builds the archives
# from the package's own schemes; `docc merge` (Xcode 16+) joins them. Needs Pages enabled with source "GitHub Actions"
# (repository Settings > Pages). Copied from CuboKit's pattern; the module list is read from the .docc catalogs.
name: Docs
Expand Down
2 changes: 1 addition & 1 deletion CHANGELOG.md
Original file line number Diff line number Diff line change
Expand Up @@ -294,6 +294,6 @@ FormatoHDF5.
## 0.3.0, 0.2.0, 0.1.0

EELS thickness (t/λ) and the C K ratio method; the EDX quantification stage gates; the first extraction from the source
application (`../_archive/SpectroKit/docs/history.md`).
application (its history is archived).

Consumer notes for 0.0.1 to 0.5.0 are archived.
Original file line number Diff line number Diff line change
Expand Up @@ -10,7 +10,7 @@
// R3 release a candidate the proposer held back as a sum-peak question is released when net / L_D >= 10, and then
// meets R1 and R2 like any other pick.
//
// The cuts (3, 10 L_D, 1.5 FWHM) were read off 78 files of ONE operator, one detector, mostly Al-Mg-Si; CLAUDE.md: a threshold is
// The cuts (3, 10 L_D, 1.5 FWHM) were read off 78 files of ONE operator, one detector, mostly Al-Mg-Si; the kit's rule: a threshold is
// a property of the dataset until measured on every dataset it will touch. `UNVALIDATED` stays on the Auto ID row; the hold-out
// and the demo-edx ladder are reported in docs/archive/v5/wp4-autoid-results-2026-10-07.md. A rule that did not hold there is
// switched off in `shipped`, not retuned (a changed cut is a new registration): R1's actinide half ships, R1's beside-K half
Expand Down
2 changes: 1 addition & 1 deletion Tests/Fixtures/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -3,7 +3,7 @@
One row per committed fixture file under `Tests/*/Fixtures/`: size, the script under `tools/` that names it (the generator or its
pin script; a few scripts write a whole directory; a script found only by file-name prefix, tool-folder name or test module is marked so), and a note where the origin is not the kit's own pinned reference. 186 files.

Licence: every fixture is generated from this kit's own code and the pinned Python reference (see `CLAUDE.md`, Reference) and ships under the kit's GPL-3.0-or-later licence, except the rows marked *third party*. Nothing here is the owner's data: the committed text fixtures carry no owner path or name (checked by search), and real datasets are read from `$MACEM_DATA` and never copied into the repo.
Licence: every fixture is generated from this kit's own code and the pinned Python reference (see `docs/parity.md`, Reference) and ships under the kit's GPL-3.0-or-later licence, except the rows marked *third party*. Nothing here is the owner's data: the committed text fixtures carry no owner path or name (checked by search), and real datasets are read from `$MACEM_DATA` and never copied into the repo.

## Third party

Expand Down
2 changes: 1 addition & 1 deletion Tests/SpectroEELSTests/Fixtures/pins.json
Original file line number Diff line number Diff line change
@@ -1,7 +1,7 @@
{
"schema": "spectrokit.eels.pins/1",
"generator": "tools/eels-thickness/pins.py",
"notTruth": "These are exSpy's answers. Agreement with exSpy is agreement, not truth (CLAUDE.md); the truth is planted-truth.json. A difference from a pin is a failure of the port, never a tolerance to widen.",
"notTruth": "These are exSpy's answers. Agreement with exSpy is agreement, not truth; the truth is planted-truth.json. A difference from a pin is a failure of the port, never a tolerance to widen.",
"reference": {
"exspyCommit": "7185a4d13d7d5c550b7bb8d1e9bd62af9cc4203e",
"exspyCheckoutHeadVerified": true,
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -2,7 +2,7 @@ import Foundation
import Testing
import SpectroCore

/// Benchmark (not a test; runs only with SPECTRO_BENCH_SPARSE=1, under tools/memcap.sh): 300 disjoint regions of a 512 x 512 scan
/// Benchmark (not a test; runs only with SPECTRO_BENCH_SPARSE=1, under a memory cap): 300 disjoint regions of a 512 x 512 scan
/// transported onto a 512 x 512 spectrum grid through a rotated, scaled registration. Dense baseline: one
/// `transport(labels:classCount: 300)`; sparse: `regions(...)`. Prints wall time (best of 3), result bytes and peak resident size.
struct BenchmarkSparseRegionsTests {
Expand Down
2 changes: 1 addition & 1 deletion Tests/SpectroTestSupport/EDXFixtures.swift
Original file line number Diff line number Diff line change
Expand Up @@ -149,7 +149,7 @@ package extension Dictionary where Key == String, Value == Any {
func s(_ k: String) -> String { self[k] as! String }
}

// MARK: parallel helper (at most 8 workers: suite CLAUDE.md, heavy CPU work)
// MARK: parallel helper (at most 8 workers: the heavy-CPU limit)

package final class ResultBox<T: Sendable>: @unchecked Sendable {
private let lock = NSLock()
Expand Down
2 changes: 1 addition & 1 deletion docs/benchmarks.md
Original file line number Diff line number Diff line change
Expand Up @@ -2,7 +2,7 @@

Written 2026-10-21 for 0.8.0. Harness: `tools/benchmarks/run.sh [small,typical]` (Swift: `Tests/SpectroEDXTests/BenchmarkQuantificationTests.swift`,
`Tests/SpectroEELSTests/BenchmarkThicknessTests.swift`, built by `swift test -c release`, enabled by `SPECTRO_BENCH_OUT`; Python: `tools/benchmarks/bench.py`),
under `tools/memcap.sh` (8 GB; measured Swift peak 1.3 GB at the typical size). Inputs are closed-form on both sides (formulas in the harness headers), float64.
under an 8 GB memory cap (measured Swift peak 1.3 GB at the typical size). Inputs are closed-form on both sides (formulas in the harness headers), float64.
One warm-up, then five runs, median and (min-max).

Machine: Apple M5 Pro, 64 GB, macOS 27.0.1 (26A434), Swift 6.4 (swiftlang-6.4.0.34.1), Python 3.12.14, NumPy 2.5.4, HyperSpy 2.4.0, exSpy 0.3.2
Expand Down
6 changes: 3 additions & 3 deletions docs/ci-parity.md
Original file line number Diff line number Diff line change
Expand Up @@ -40,7 +40,7 @@ Five golden first differences:
## Backend constraint

The pinned SciPy wheel itself links Accelerate (checked with the pinned venv's `scipy.show_config()`, under
`../tools/memcap.sh`). The logged coefficient arrays show a CPU/toolchain-dependent small residual at a mathematically zero coefficient.
a memory cap). The logged coefficient arrays show a CPU/toolchain-dependent small residual at a mathematically zero coefficient.
`ElasticThreshold` compares derivative magnitudes with an exact `<=`, then chooses a discrete channel. The observed tiny
coefficient differences can therefore produce large changes in thickness; the reduction to only edge mismatches when
pinned coefficients are supplied isolates the coefficient solve as the main suspected cause. A more accurate scalar solver alone does not
Expand All @@ -54,13 +54,13 @@ The SG coefficients and edge fits stay on LAPACK's SVD (`dgelsd`), as in SciPy's
retain their exact comparisons. The selected threshold channel remains an exact assertion against the new policy oracle.

`tools/eels-thickness/make_portable_thresholds.py` runs pinned SciPy 1.18.1, NumPy 2.5.4 and HyperSpy 2.4.0 in
`../References/python/hyperspy` under `../tools/memcap.sh`. It applies the declared tie policy to SciPy's derivatives, retaining the
`../References/python/hyperspy` under a memory cap. It applies the declared tie policy to SciPy's derivatives, retaining the
raw exSpy `7185a4d1` answers alongside it. Its committed `portable-thresholds.json` is about 42 KB: 18 routes / 138 pixels,
with 20 deliberately changed thresholds. It records input SHA256s, raw/selected thresholds, derivative minima, slack, I0,
It and t/lambda. Run from this repository:

```sh
../tools/memcap.sh -- ../References/python/hyperspy/bin/python -I -B tools/eels-thickness/make_portable_thresholds.py
../References/python/hyperspy/bin/python -I -B tools/eels-thickness/make_portable_thresholds.py
```

The original SG coefficient and derivative fixtures remain unchanged. Coefficients are bounded at `1e-13 * coefficient L1`,
Expand Down
4 changes: 2 additions & 2 deletions docs/design/eels-task-16.md
Original file line number Diff line number Diff line change
Expand Up @@ -3,7 +3,7 @@
Suite ROADMAP task 16, SpectroKit 0.9.0. Written 2026-10-21 for a Mac session to implement without further design work.
Reference: exSpy 0.3.2 (`exspy/signals/eels.py`, `exspy/models/eelsmodel.py`, `exspy/components/eels_cl_edge.py`) and
HyperSpy 2.4.0 (`hyperspy/_signals/signal1d.py`, `hyperspy/misc/math_tools.py`, `hyperspy/_components/power_law.py`)
in `../References/python/hyperspy` (suite `tools/setup-python-refs.sh hyperspy`; NumPy 2.5.4, SciPy 1.18.1). Line
in `../References/python/hyperspy` (NumPy 2.5.4, SciPy 1.18.1). Line
numbers below are from those files. Every result stays `validation: "none"`. No code beyond signatures; one screen per
section.

Expand Down Expand Up @@ -346,6 +346,6 @@ Bluestein in SpectroKit was the alternative and is not worth a copy.
7. The bundled L edges (Mg, Al, Si, Fe, Cu) are an engineering pick for the study's alloys; a different list is a
one-line change in `extract_gos.py` plus a resources commit.

Closeout for the implementing session (CLAUDE.md): fast target green per commit, full suite before the tag, `CHANGELOG.md`
Closeout for the implementing session: fast target green per commit, full suite before the tag, `CHANGELOG.md`
0.9.0 entry with a Migration paragraph (new dependency, `CoreLossEdge.subshell` values beyond `K1`), `docs/parity.md`
rows per section 9, `docs/status.md` one line, open item 4 rewritten, hand-off paragraph in the last commit.
2 changes: 1 addition & 1 deletion docs/open-items.md
Original file line number Diff line number Diff line change
@@ -1,6 +1,6 @@
# Open items

Slowest against Python first (diagnose before changing a number, `../CLAUDE.md`).
Slowest against Python first (diagnose before changing a number).

1. **t/lambda with a fixed threshold is 2x slower than HyperSpy at 64 x 64 x 2048** (9.1 ms against 4.5 ms, `docs/benchmarks.md`). Hypothesis
to test, not yet tested: the per-pixel pairwise sums (bit-faithful to NumPy, required by the pins) and the zero-loss-centre argmax run single-threaded,
Expand Down
2 changes: 1 addition & 1 deletion docs/parity.md
Original file line number Diff line number Diff line change
Expand Up @@ -4,7 +4,7 @@ What SpectroKit has of exSpy's and HyperSpy's EDS and EELS API, row by row, what
source on disk on 2026-10-21 for 0.8.0 (the walk of every public method of `exspy/signals/{eds,eds_tem,eds_sem,eels,dielectric_function}.py`,
`exspy/models/*`, `exspy/components/*`, `exspy/utils/*`, `exspy/material.py`, and the tests in `exspy/tests/` for tolerances).

**Reference.** exSpy 0.3.2 and HyperSpy 2.4.0 in `References/python/hyperspy` (the suite folder's gitignored Python reference, `tools/setup-python-refs.sh`);
**Reference.** exSpy 0.3.2 and HyperSpy 2.4.0 in `References/python/hyperspy` (a gitignored Python reference beside the kit);
fixtures stay pinned to exSpy `7185a4d1` (`References/exspy-7185a4d1`) with RosettaSciIO 0.14.0, NumPy 2.5.4, SciPy 1.18.1, except the
0.8.0 maps fixture, which is exSpy 0.3.2's own output (`tools/edx-quant-maps/pins.py`). The public methods were read from the installed
modules; the tolerance column is what exSpy's own test files assert (`assert_allclose` defaults are rtol 1e-7).
Expand Down
7 changes: 4 additions & 3 deletions tools/benchmarks/run.sh
Original file line number Diff line number Diff line change
Expand Up @@ -6,7 +6,8 @@ set -eu
cd "$(dirname "$0")/../.."
sizes="${1:-small,typical}"
out="$PWD/.build/benchmarks"; mkdir -p "$out"
memcap="$PWD/../tools/memcap.sh"
SPECTRO_BENCH_OUT="$out" SPECTRO_BENCH_SIZES="$sizes" "$memcap" -- swift test -c release --filter 'Benchmark' >"$out/swift.log" 2>&1
"$memcap" -- ../refs/hyperspy/bin/python -I -B tools/benchmarks/bench.py "$out" "$sizes" >"$out/python.log" 2>&1
# SPECTRO_BENCH_CAP: a memory-cap wrapper run as `<cap> -- <command>` (8 GB); the benchmark never runs uncapped.
cap="${SPECTRO_BENCH_CAP:?set SPECTRO_BENCH_CAP to a memory-cap wrapper}"
SPECTRO_BENCH_OUT="$out" SPECTRO_BENCH_SIZES="$sizes" "$cap" -- swift test -c release --filter 'Benchmark' >"$out/swift.log" 2>&1
"$cap" -- ../refs/hyperspy/bin/python -I -B tools/benchmarks/bench.py "$out" "$sizes" >"$out/python.log" 2>&1
../refs/hyperspy/bin/python -I -B tools/benchmarks/table.py "$out"
2 changes: 1 addition & 1 deletion tools/edx-stages/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -7,7 +7,7 @@ SciPy 1.18.1, h5py 3.16.0), run `-I -B`. Fixtures: `Tests/SpectroEDXTests/Fixtur
1. `../References/python/hyperspy/bin/python -I -B tools/edx-stages/make_planted.py` (NumPy only; 0.4 s) -> `edx-planted*.{json,f64,i16}`, `edx-calibration*`.
2. `../References/python/hyperspy/bin/python -I -B tools/edx-stages/make_velox_streams.py` (h5py) -> `velox-stream-*.emd`, `velox-streams-truth.json`.
3. `../References/python/hyperspy/bin/python -I -B tools/edx-stages/pins.py` (about 110 s) -> `pins.json`, `edx-axis-e2.f64`, `edx-pins-intensity.f64`.
4. `... tools/edx-stages/findings.py` printed the numbers below (not a fixture writer; archived 2026-10-15 to `../_archive/SpectroKit/tools/edx-stages/findings.py`).
4. `... tools/edx-stages/findings.py` printed the numbers below (not a fixture writer; archived 2026-10-15).
Docstrings of each script define every parameter; this file lists readings made and findings.

## Formats the Swift tests read
Expand Down
4 changes: 2 additions & 2 deletions tools/eels-thickness/make_portable_thresholds.py
Original file line number Diff line number Diff line change
@@ -1,8 +1,8 @@
#!/usr/bin/env python3
"""Pinned SciPy derivatives plus SpectroKit's documented numerical-tie policy.

Run from SpectroKit (never on CI):
../tools/memcap.sh -- ../References/python/hyperspy/bin/python -I -B tools/eels-thickness/make_portable_thresholds.py
Run from SpectroKit (never on CI), under a memory cap:
../References/python/hyperspy/bin/python -I -B tools/eels-thickness/make_portable_thresholds.py
Retains all raw exSpy/SciPy fixtures. This is a deliberate policy deviation, not
an assertion that raw exSpy chooses the same channels on different CPUs.
"""
Expand Down
2 changes: 1 addition & 1 deletion tools/eels-thickness/pins.py
Original file line number Diff line number Diff line change
Expand Up @@ -632,7 +632,7 @@ def main():
pins = {
"schema": "spectrokit.eels.pins/1",
"generator": "tools/eels-thickness/pins.py",
"notTruth": "These are exSpy's answers. Agreement with exSpy is agreement, not truth (CLAUDE.md); "
"notTruth": "These are exSpy's answers. Agreement with exSpy is agreement, not truth; "
"the truth is planted-truth.json. A difference from a pin is a failure of the port, never a tolerance to widen.",
"reference": {"exspyCommit": EXSPY_COMMIT, "exspyCheckoutHeadVerified": commit_ok,
"hyperspy": hyperspy.__version__, "rsciio": rsciio.__version__, "numpy": np.__version__,
Expand Down
Loading