diff --git a/.github/dependabot.yml b/.github/dependabot.yml new file mode 100644 index 0000000..9e70900 --- /dev/null +++ b/.github/dependabot.yml @@ -0,0 +1,11 @@ +version: 2 +updates: + # GitHub Actions only, monthly, one grouped PR; it merges itself when CI is green (dependabot-auto-merge.yml). + # Swift packages stay out: kit and app pins are exact and moved on purpose. + - package-ecosystem: github-actions + directory: / + schedule: + interval: monthly + groups: + github-actions: + patterns: ["*"] diff --git a/.github/workflows/ci.yml b/.github/workflows/ci.yml index 0f33cf6..83a8e41 100644 --- a/.github/workflows/ci.yml +++ b/.github/workflows/ci.yml @@ -69,3 +69,14 @@ jobs: swift --version | grep -E "Swift version 6\.([2-9]|[1-9][0-9])" || { echo "Swift 6.2 or newer needed"; exit 1; } - name: Gate (tools/test.sh, release) run: tools/test.sh + + private-refs: + name: No private references + runs-on: ubuntu-latest + steps: + - uses: actions/checkout@v4 + - name: Scan tracked files + run: | + if git grep -n -I -E '_archive|CLAUDE\.md|memcap|macEM-suite|setup-python-refs' -- ':!.gitignore' ':!.github/workflows/ci.yml'; then + echo "Private references above; this repo is public."; exit 1 + fi diff --git a/.github/workflows/docs.yml b/.github/workflows/docs.yml index 81ced30..714acfa 100644 --- a/.github/workflows/docs.yml +++ b/.github/workflows/docs.yml @@ -2,7 +2,7 @@ # published to GitHub Pages at https://mac4dstem.github.io/SpectroKit/documentation/ on every tag. # On a pull request the site is built and uploaded as an artifact only, so a broken catalog fails before the tag. # -# No swift-docc-plugin: the kit takes no package dependencies (CLAUDE.md). `xcodebuild docbuild` builds the archives +# No swift-docc-plugin: the kit takes no package dependencies. `xcodebuild docbuild` builds the archives # from the package's own schemes; `docc merge` (Xcode 16+) joins them. Needs Pages enabled with source "GitHub Actions" # (repository Settings > Pages). Copied from CuboKit's pattern; the module list is read from the .docc catalogs. name: Docs diff --git a/CHANGELOG.md b/CHANGELOG.md index e421305..659b3eb 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -294,6 +294,6 @@ FormatoHDF5. ## 0.3.0, 0.2.0, 0.1.0 EELS thickness (t/λ) and the C K ratio method; the EDX quantification stage gates; the first extraction from the source -application (`../_archive/SpectroKit/docs/history.md`). +application (its history is archived). Consumer notes for 0.0.1 to 0.5.0 are archived. diff --git a/Sources/SpectroEDX/Spectroscopy/Proposer/ProposalRules.swift b/Sources/SpectroEDX/Spectroscopy/Proposer/ProposalRules.swift index 8f23770..97faf0b 100644 --- a/Sources/SpectroEDX/Spectroscopy/Proposer/ProposalRules.swift +++ b/Sources/SpectroEDX/Spectroscopy/Proposer/ProposalRules.swift @@ -10,7 +10,7 @@ // R3 release a candidate the proposer held back as a sum-peak question is released when net / L_D >= 10, and then // meets R1 and R2 like any other pick. // -// The cuts (3, 10 L_D, 1.5 FWHM) were read off 78 files of ONE operator, one detector, mostly Al-Mg-Si; CLAUDE.md: a threshold is +// The cuts (3, 10 L_D, 1.5 FWHM) were read off 78 files of ONE operator, one detector, mostly Al-Mg-Si; the kit's rule: a threshold is // a property of the dataset until measured on every dataset it will touch. `UNVALIDATED` stays on the Auto ID row; the hold-out // and the demo-edx ladder are reported in docs/archive/v5/wp4-autoid-results-2026-10-07.md. A rule that did not hold there is // switched off in `shipped`, not retuned (a changed cut is a new registration): R1's actinide half ships, R1's beside-K half diff --git a/Tests/Fixtures/README.md b/Tests/Fixtures/README.md index 703cc0c..b5064e6 100644 --- a/Tests/Fixtures/README.md +++ b/Tests/Fixtures/README.md @@ -3,7 +3,7 @@ One row per committed fixture file under `Tests/*/Fixtures/`: size, the script under `tools/` that names it (the generator or its pin script; a few scripts write a whole directory; a script found only by file-name prefix, tool-folder name or test module is marked so), and a note where the origin is not the kit's own pinned reference. 186 files. -Licence: every fixture is generated from this kit's own code and the pinned Python reference (see `CLAUDE.md`, Reference) and ships under the kit's GPL-3.0-or-later licence, except the rows marked *third party*. Nothing here is the owner's data: the committed text fixtures carry no owner path or name (checked by search), and real datasets are read from `$MACEM_DATA` and never copied into the repo. +Licence: every fixture is generated from this kit's own code and the pinned Python reference (see `docs/parity.md`, Reference) and ships under the kit's GPL-3.0-or-later licence, except the rows marked *third party*. Nothing here is the owner's data: the committed text fixtures carry no owner path or name (checked by search), and real datasets are read from `$MACEM_DATA` and never copied into the repo. ## Third party diff --git a/Tests/SpectroEELSTests/Fixtures/pins.json b/Tests/SpectroEELSTests/Fixtures/pins.json index cd5c04c..4c49300 100644 --- a/Tests/SpectroEELSTests/Fixtures/pins.json +++ b/Tests/SpectroEELSTests/Fixtures/pins.json @@ -1,7 +1,7 @@ { "schema": "spectrokit.eels.pins/1", "generator": "tools/eels-thickness/pins.py", - "notTruth": "These are exSpy's answers. Agreement with exSpy is agreement, not truth (CLAUDE.md); the truth is planted-truth.json. A difference from a pin is a failure of the port, never a tolerance to widen.", + "notTruth": "These are exSpy's answers. Agreement with exSpy is agreement, not truth; the truth is planted-truth.json. A difference from a pin is a failure of the port, never a tolerance to widen.", "reference": { "exspyCommit": "7185a4d13d7d5c550b7bb8d1e9bd62af9cc4203e", "exspyCheckoutHeadVerified": true, diff --git a/Tests/SpectroKitParityTests/BenchmarkSparseRegionsTests.swift b/Tests/SpectroKitParityTests/BenchmarkSparseRegionsTests.swift index 66547fc..b7c8ad8 100644 --- a/Tests/SpectroKitParityTests/BenchmarkSparseRegionsTests.swift +++ b/Tests/SpectroKitParityTests/BenchmarkSparseRegionsTests.swift @@ -2,7 +2,7 @@ import Foundation import Testing import SpectroCore -/// Benchmark (not a test; runs only with SPECTRO_BENCH_SPARSE=1, under tools/memcap.sh): 300 disjoint regions of a 512 x 512 scan +/// Benchmark (not a test; runs only with SPECTRO_BENCH_SPARSE=1, under a memory cap): 300 disjoint regions of a 512 x 512 scan /// transported onto a 512 x 512 spectrum grid through a rotated, scaled registration. Dense baseline: one /// `transport(labels:classCount: 300)`; sparse: `regions(...)`. Prints wall time (best of 3), result bytes and peak resident size. struct BenchmarkSparseRegionsTests { diff --git a/Tests/SpectroTestSupport/EDXFixtures.swift b/Tests/SpectroTestSupport/EDXFixtures.swift index 50e9354..4ba5a4e 100644 --- a/Tests/SpectroTestSupport/EDXFixtures.swift +++ b/Tests/SpectroTestSupport/EDXFixtures.swift @@ -149,7 +149,7 @@ package extension Dictionary where Key == String, Value == Any { func s(_ k: String) -> String { self[k] as! String } } -// MARK: parallel helper (at most 8 workers: suite CLAUDE.md, heavy CPU work) +// MARK: parallel helper (at most 8 workers: the heavy-CPU limit) package final class ResultBox: @unchecked Sendable { private let lock = NSLock() diff --git a/docs/benchmarks.md b/docs/benchmarks.md index 060f1c3..903eaa6 100644 --- a/docs/benchmarks.md +++ b/docs/benchmarks.md @@ -2,7 +2,7 @@ Written 2026-10-21 for 0.8.0. Harness: `tools/benchmarks/run.sh [small,typical]` (Swift: `Tests/SpectroEDXTests/BenchmarkQuantificationTests.swift`, `Tests/SpectroEELSTests/BenchmarkThicknessTests.swift`, built by `swift test -c release`, enabled by `SPECTRO_BENCH_OUT`; Python: `tools/benchmarks/bench.py`), -under `tools/memcap.sh` (8 GB; measured Swift peak 1.3 GB at the typical size). Inputs are closed-form on both sides (formulas in the harness headers), float64. +under an 8 GB memory cap (measured Swift peak 1.3 GB at the typical size). Inputs are closed-form on both sides (formulas in the harness headers), float64. One warm-up, then five runs, median and (min-max). Machine: Apple M5 Pro, 64 GB, macOS 27.0.1 (26A434), Swift 6.4 (swiftlang-6.4.0.34.1), Python 3.12.14, NumPy 2.5.4, HyperSpy 2.4.0, exSpy 0.3.2 diff --git a/docs/ci-parity.md b/docs/ci-parity.md index ff514d8..e720157 100644 --- a/docs/ci-parity.md +++ b/docs/ci-parity.md @@ -40,7 +40,7 @@ Five golden first differences: ## Backend constraint The pinned SciPy wheel itself links Accelerate (checked with the pinned venv's `scipy.show_config()`, under -`../tools/memcap.sh`). The logged coefficient arrays show a CPU/toolchain-dependent small residual at a mathematically zero coefficient. +a memory cap). The logged coefficient arrays show a CPU/toolchain-dependent small residual at a mathematically zero coefficient. `ElasticThreshold` compares derivative magnitudes with an exact `<=`, then chooses a discrete channel. The observed tiny coefficient differences can therefore produce large changes in thickness; the reduction to only edge mismatches when pinned coefficients are supplied isolates the coefficient solve as the main suspected cause. A more accurate scalar solver alone does not @@ -54,13 +54,13 @@ The SG coefficients and edge fits stay on LAPACK's SVD (`dgelsd`), as in SciPy's retain their exact comparisons. The selected threshold channel remains an exact assertion against the new policy oracle. `tools/eels-thickness/make_portable_thresholds.py` runs pinned SciPy 1.18.1, NumPy 2.5.4 and HyperSpy 2.4.0 in -`../References/python/hyperspy` under `../tools/memcap.sh`. It applies the declared tie policy to SciPy's derivatives, retaining the +`../References/python/hyperspy` under a memory cap. It applies the declared tie policy to SciPy's derivatives, retaining the raw exSpy `7185a4d1` answers alongside it. Its committed `portable-thresholds.json` is about 42 KB: 18 routes / 138 pixels, with 20 deliberately changed thresholds. It records input SHA256s, raw/selected thresholds, derivative minima, slack, I0, It and t/lambda. Run from this repository: ```sh -../tools/memcap.sh -- ../References/python/hyperspy/bin/python -I -B tools/eels-thickness/make_portable_thresholds.py +../References/python/hyperspy/bin/python -I -B tools/eels-thickness/make_portable_thresholds.py ``` The original SG coefficient and derivative fixtures remain unchanged. Coefficients are bounded at `1e-13 * coefficient L1`, diff --git a/docs/design/eels-task-16.md b/docs/design/eels-task-16.md index f0c1737..3371cc7 100644 --- a/docs/design/eels-task-16.md +++ b/docs/design/eels-task-16.md @@ -3,7 +3,7 @@ Suite ROADMAP task 16, SpectroKit 0.9.0. Written 2026-10-21 for a Mac session to implement without further design work. Reference: exSpy 0.3.2 (`exspy/signals/eels.py`, `exspy/models/eelsmodel.py`, `exspy/components/eels_cl_edge.py`) and HyperSpy 2.4.0 (`hyperspy/_signals/signal1d.py`, `hyperspy/misc/math_tools.py`, `hyperspy/_components/power_law.py`) -in `../References/python/hyperspy` (suite `tools/setup-python-refs.sh hyperspy`; NumPy 2.5.4, SciPy 1.18.1). Line +in `../References/python/hyperspy` (NumPy 2.5.4, SciPy 1.18.1). Line numbers below are from those files. Every result stays `validation: "none"`. No code beyond signatures; one screen per section. @@ -346,6 +346,6 @@ Bluestein in SpectroKit was the alternative and is not worth a copy. 7. The bundled L edges (Mg, Al, Si, Fe, Cu) are an engineering pick for the study's alloys; a different list is a one-line change in `extract_gos.py` plus a resources commit. -Closeout for the implementing session (CLAUDE.md): fast target green per commit, full suite before the tag, `CHANGELOG.md` +Closeout for the implementing session: fast target green per commit, full suite before the tag, `CHANGELOG.md` 0.9.0 entry with a Migration paragraph (new dependency, `CoreLossEdge.subshell` values beyond `K1`), `docs/parity.md` rows per section 9, `docs/status.md` one line, open item 4 rewritten, hand-off paragraph in the last commit. diff --git a/docs/open-items.md b/docs/open-items.md index 4c72ff9..b6368ef 100644 --- a/docs/open-items.md +++ b/docs/open-items.md @@ -1,6 +1,6 @@ # Open items -Slowest against Python first (diagnose before changing a number, `../CLAUDE.md`). +Slowest against Python first (diagnose before changing a number). 1. **t/lambda with a fixed threshold is 2x slower than HyperSpy at 64 x 64 x 2048** (9.1 ms against 4.5 ms, `docs/benchmarks.md`). Hypothesis to test, not yet tested: the per-pixel pairwise sums (bit-faithful to NumPy, required by the pins) and the zero-loss-centre argmax run single-threaded, diff --git a/docs/parity.md b/docs/parity.md index 3a17bf0..d4ebc0d 100644 --- a/docs/parity.md +++ b/docs/parity.md @@ -4,7 +4,7 @@ What SpectroKit has of exSpy's and HyperSpy's EDS and EELS API, row by row, what source on disk on 2026-10-21 for 0.8.0 (the walk of every public method of `exspy/signals/{eds,eds_tem,eds_sem,eels,dielectric_function}.py`, `exspy/models/*`, `exspy/components/*`, `exspy/utils/*`, `exspy/material.py`, and the tests in `exspy/tests/` for tolerances). -**Reference.** exSpy 0.3.2 and HyperSpy 2.4.0 in `References/python/hyperspy` (the suite folder's gitignored Python reference, `tools/setup-python-refs.sh`); +**Reference.** exSpy 0.3.2 and HyperSpy 2.4.0 in `References/python/hyperspy` (a gitignored Python reference beside the kit); fixtures stay pinned to exSpy `7185a4d1` (`References/exspy-7185a4d1`) with RosettaSciIO 0.14.0, NumPy 2.5.4, SciPy 1.18.1, except the 0.8.0 maps fixture, which is exSpy 0.3.2's own output (`tools/edx-quant-maps/pins.py`). The public methods were read from the installed modules; the tolerance column is what exSpy's own test files assert (`assert_allclose` defaults are rtol 1e-7). diff --git a/tools/benchmarks/run.sh b/tools/benchmarks/run.sh index 5a1da06..df71f58 100755 --- a/tools/benchmarks/run.sh +++ b/tools/benchmarks/run.sh @@ -6,7 +6,8 @@ set -eu cd "$(dirname "$0")/../.." sizes="${1:-small,typical}" out="$PWD/.build/benchmarks"; mkdir -p "$out" -memcap="$PWD/../tools/memcap.sh" -SPECTRO_BENCH_OUT="$out" SPECTRO_BENCH_SIZES="$sizes" "$memcap" -- swift test -c release --filter 'Benchmark' >"$out/swift.log" 2>&1 -"$memcap" -- ../refs/hyperspy/bin/python -I -B tools/benchmarks/bench.py "$out" "$sizes" >"$out/python.log" 2>&1 +# SPECTRO_BENCH_CAP: a memory-cap wrapper run as ` -- ` (8 GB); the benchmark never runs uncapped. +cap="${SPECTRO_BENCH_CAP:?set SPECTRO_BENCH_CAP to a memory-cap wrapper}" +SPECTRO_BENCH_OUT="$out" SPECTRO_BENCH_SIZES="$sizes" "$cap" -- swift test -c release --filter 'Benchmark' >"$out/swift.log" 2>&1 +"$cap" -- ../refs/hyperspy/bin/python -I -B tools/benchmarks/bench.py "$out" "$sizes" >"$out/python.log" 2>&1 ../refs/hyperspy/bin/python -I -B tools/benchmarks/table.py "$out" diff --git a/tools/edx-stages/README.md b/tools/edx-stages/README.md index 75adaa3..4665628 100644 --- a/tools/edx-stages/README.md +++ b/tools/edx-stages/README.md @@ -7,7 +7,7 @@ SciPy 1.18.1, h5py 3.16.0), run `-I -B`. Fixtures: `Tests/SpectroEDXTests/Fixtur 1. `../References/python/hyperspy/bin/python -I -B tools/edx-stages/make_planted.py` (NumPy only; 0.4 s) -> `edx-planted*.{json,f64,i16}`, `edx-calibration*`. 2. `../References/python/hyperspy/bin/python -I -B tools/edx-stages/make_velox_streams.py` (h5py) -> `velox-stream-*.emd`, `velox-streams-truth.json`. 3. `../References/python/hyperspy/bin/python -I -B tools/edx-stages/pins.py` (about 110 s) -> `pins.json`, `edx-axis-e2.f64`, `edx-pins-intensity.f64`. -4. `... tools/edx-stages/findings.py` printed the numbers below (not a fixture writer; archived 2026-10-15 to `../_archive/SpectroKit/tools/edx-stages/findings.py`). +4. `... tools/edx-stages/findings.py` printed the numbers below (not a fixture writer; archived 2026-10-15). Docstrings of each script define every parameter; this file lists readings made and findings. ## Formats the Swift tests read diff --git a/tools/eels-thickness/make_portable_thresholds.py b/tools/eels-thickness/make_portable_thresholds.py index ef18421..f65fe57 100644 --- a/tools/eels-thickness/make_portable_thresholds.py +++ b/tools/eels-thickness/make_portable_thresholds.py @@ -1,8 +1,8 @@ #!/usr/bin/env python3 """Pinned SciPy derivatives plus SpectroKit's documented numerical-tie policy. -Run from SpectroKit (never on CI): -../tools/memcap.sh -- ../References/python/hyperspy/bin/python -I -B tools/eels-thickness/make_portable_thresholds.py +Run from SpectroKit (never on CI), under a memory cap: +../References/python/hyperspy/bin/python -I -B tools/eels-thickness/make_portable_thresholds.py Retains all raw exSpy/SciPy fixtures. This is a deliberate policy deviation, not an assertion that raw exSpy chooses the same channels on different CPUs. """ diff --git a/tools/eels-thickness/pins.py b/tools/eels-thickness/pins.py index e0973cd..2cd8eed 100755 --- a/tools/eels-thickness/pins.py +++ b/tools/eels-thickness/pins.py @@ -632,7 +632,7 @@ def main(): pins = { "schema": "spectrokit.eels.pins/1", "generator": "tools/eels-thickness/pins.py", - "notTruth": "These are exSpy's answers. Agreement with exSpy is agreement, not truth (CLAUDE.md); " + "notTruth": "These are exSpy's answers. Agreement with exSpy is agreement, not truth; " "the truth is planted-truth.json. A difference from a pin is a failure of the port, never a tolerance to widen.", "reference": {"exspyCommit": EXSPY_COMMIT, "exspyCheckoutHeadVerified": commit_ok, "hyperspy": hyperspy.__version__, "rsciio": rsciio.__version__, "numpy": np.__version__,