From dc5c096fa528057ada4aa92c47c21f3e2354f9c7 Mon Sep 17 00:00:00 2001 From: Claude Date: Wed, 28 Oct 2026 15:06:52 +0000 Subject: [PATCH] docs: list the 0.9.0 public names in DocC and the featurization gaps The CuboBragg, CuboCore, CuboEmbedding and CuboVirtual landing pages gain Topics groups for every public type they lacked (polar cube, radial background, resident cube, ellipse fit, calibration and cube editing, Bragg-vector classification and virtual images, featurization, NMF, Gaussian mixtures). CuboEmbedding's Scope and docs/embedding.md name what is not ported from py4DSTEM's Featurization. Done: DocC and text half of CuboKit#15. Next: exercise the new names in tools/consumer-check on a Mac. Co-Authored-By: Claude Opus 5.5 Claude-Session: https://claude.ai/code/session_01VkKZEVGZzxyb2accSwvsbj --- Sources/CuboBragg/CuboBragg.docc/CuboBragg.md | 15 +++++++++ Sources/CuboCore/CuboCore.docc/CuboCore.md | 31 +++++++++++++++++++ .../CuboEmbedding.docc/CuboEmbedding.md | 22 ++++++++++++- .../CuboEmbedding/CuboEmbedding.docc/Scope.md | 13 ++++++-- .../CuboVirtual.docc/CuboVirtual.md | 18 +++++++++++ docs/embedding.md | 15 ++++++++- docs/open-items.md | 2 +- 7 files changed, 110 insertions(+), 6 deletions(-) diff --git a/Sources/CuboBragg/CuboBragg.docc/CuboBragg.md b/Sources/CuboBragg/CuboBragg.docc/CuboBragg.md index 1e131f9..24f9dd1 100644 --- a/Sources/CuboBragg/CuboBragg.docc/CuboBragg.md +++ b/Sources/CuboBragg/CuboBragg.docc/CuboBragg.md @@ -22,6 +22,11 @@ is the median fill of non-finite pixels, public. ``BraggDiskOptions/detectorAdapted(detectorRows:detectorColumns:probeRadius:base:)``. ``ProbeRingHint`` is the app's suggestion for a ring-structured probe's radius. +``BraggVectorClassification`` is py4DSTEM's `braggvectorclassification`: peaks labelled by the nearest reference vector, +an NMF of the label matrix, then split, merge, accept and reject steps, and the class images and class patterns. +``BraggVectors/virtualImage(detector:)`` sums the peaks inside a ``BraggVirtualDetector`` at each scan position +(py4DSTEM's `get_virtual_image`) into a ``BraggVirtualImage``. + ## Topics ### Essentials @@ -49,6 +54,16 @@ is the median fill of non-finite pixels, public. - ``BraggNonFinitePixels`` - ``BraggPrecision`` +### Classification + +- ``BraggVectorClassification`` + +### Virtual images from Bragg vectors + +- ``BraggVirtualImage`` +- ``BraggVirtualDetector`` +- ``BraggDetectorCenter`` + ### Probe - ``ProbeRingHint`` diff --git a/Sources/CuboCore/CuboCore.docc/CuboCore.md b/Sources/CuboCore/CuboCore.docc/CuboCore.md index b6e0e16..5f1a5c5 100644 --- a/Sources/CuboCore/CuboCore.docc/CuboCore.md +++ b/Sources/CuboCore/CuboCore.docc/CuboCore.md @@ -3,6 +3,11 @@ The cube model every other module reads: a stack of diffraction patterns, one per scan position, held in memory or memory-mapped from a NumPy `.npy` file, read as Float32 and tiled along the scan. +``Calibration`` carries py4DSTEM's calibration of a cube (pixel sizes, units, origin, ellipse, QR rotation and flip, probe); +``CubeEditing`` is py4DSTEM's reshaping of a `DataCube` (scan binning and thinning, padding, axis swaps, detector resampling including the Fourier resample, the masked-pixel median filter) and +returns an ``EditedCube`` with the calibration that goes with it. ``EllipseFit`` is py4DSTEM's +`process/calibration/ellipse.py`: the algebraic 1D fit, the amorphous-ring fit and the degenerate-ellipse constraint. + ## Topics ### Articles @@ -35,6 +40,32 @@ memory-mapped from a NumPy `.npy` file, read as Float32 and tiled along the scan - ``NPYFile`` - ``NPYHeader`` +### Calibration + +- ``Calibration`` +- ``CalibrationError`` +- ``CalibrationMeasurement`` +- ``DetectorUnits`` +- ``EllipseCalibration`` +- ``ScanField`` +- ``CoordinateGrid`` + +### Editing a cube + +- ``CubeEditing`` +- ``EditedCube`` +- ``FourierResampleSpecification`` +- ``FourierResampleOptions`` +- ``FrameTransform`` + +### Ellipse fits + +- ``EllipseFit`` +- ``EllipseFitPlane`` +- ``EllipseParameters`` +- ``EllipseFitResult`` +- ``EllipseFitError`` + ### Origin fits - ``OriginFit`` diff --git a/Sources/CuboEmbedding/CuboEmbedding.docc/CuboEmbedding.md b/Sources/CuboEmbedding/CuboEmbedding.docc/CuboEmbedding.md index 250e867..801c91f 100644 --- a/Sources/CuboEmbedding/CuboEmbedding.docc/CuboEmbedding.md +++ b/Sources/CuboEmbedding/CuboEmbedding.docc/CuboEmbedding.md @@ -1,6 +1,7 @@ # ``CuboEmbedding`` -CPU principal component analysis and k-means, with scikit-learn behaviour as the primary API. +CPU principal component analysis, k-means, NMF and Gaussian mixtures, with scikit-learn behaviour as the primary API, +and py4DSTEM's featurization on top of them. Use ``PCA/fit(_:rows:columns:components:variant:)`` and ``KMeans/fit(_:rows:columns:initialCentroids:maxIterations:tolerance:isCancelled:)`` for scikit-learn 1.4.2 conventions. ``PCA/Variant/diffractionEmbeddingApp``, ``KMeans/fitSeeded(_:rows:columns:groups:seed:maxIterations:isCancelled:)`` @@ -12,6 +13,11 @@ For data too large to hold, ``PCACovarianceAccumulator`` streams the mean and co signs from a second pass. ``DiffractionFeatures/binnedVector(pattern:rows:columns:binnedSize:implementation:)`` has an Accelerate implementation, and ``KMeans`` takes an `isCancelled` closure. +``Featurization`` is py4DSTEM's `process/classification/featurization.py` on a ``FeatureMatrix`` (one row per scan +position): PCA, iterative NMF, Gaussian mixtures, scaling, feature reduction and class patterns. It rests on ``NMF`` and +``GaussianMixture`` (scikit-learn 1.9.1) and on ``NumPyRandomState``, NumPy's legacy `RandomState`, bit identical, so +seeded runs match py4DSTEM's. What is not ported is listed in . + ## Topics ### Essentials @@ -38,3 +44,17 @@ Accelerate implementation, and ``KMeans`` takes an `isCancelled` closure. - ``DiffractionFeatures`` - ``DiffractionFeatures/Implementation`` + +### Featurization + +- ``Featurization`` +- ``FeatureMatrix`` +- ``FeatureReduction`` +- ``FeaturizationNMFResult`` +- ``FeaturizationError`` + +### NMF and Gaussian mixtures + +- ``NMF`` +- ``GaussianMixture`` +- ``NumPyRandomState`` diff --git a/Sources/CuboEmbedding/CuboEmbedding.docc/Scope.md b/Sources/CuboEmbedding/CuboEmbedding.docc/Scope.md index 2764c00..4e9bf51 100644 --- a/Sources/CuboEmbedding/CuboEmbedding.docc/Scope.md +++ b/Sources/CuboEmbedding/CuboEmbedding.docc/Scope.md @@ -1,10 +1,17 @@ # Scope -``PCA`` and ``KMeans`` operate on row-major Double matrices supplied by the caller. Feature extraction from a cube, -settings and progress are not part of this module (the covariance can be streamed with ``PCACovarianceAccumulator``); ``DiffractionFeatures`` only turns one pattern into one -vector. +``PCA``, ``KMeans``, ``NMF`` and ``GaussianMixture`` operate on row-major Double matrices supplied by the caller. +Feature extraction from a cube, settings and progress are not part of this module (the covariance can be streamed with +``PCACovarianceAccumulator``); ``DiffractionFeatures`` only turns one pattern into one vector, and ``Featurization`` starts +from a ``FeatureMatrix`` the caller builds. Primary behaviour is pinned to scikit-learn 1.4.2: the covariance route to PCA with its `svd_flip` sign convention, and Lloyd's algorithm from caller-supplied centroids. mac4DSTEM's Float32 PCA, seeded k-means++ and feature binning are named options pinned to its original source. Not provided: whitening, other solvers, random or k-means++ seeding on the scikit-learn path, sample weights, prediction of new rows. See `docs/embedding.md`. + +``Featurization`` follows py4DSTEM 0.14.19 with scikit-learn 1.9.1 numbers. Not ported: `Featurization.from_braggvectors` +(it needs the Bragg types, and this module does not depend on CuboBragg), `ICA`, `RobustScaler`, `spatial_separation`, +`consensus`, `add_features` and `delete_features`. The `nmf` and `gmm` branches of `get_class_DPs` are buggy upstream and +are not copied; ``Featurization/classPatterns(weights:patterns:pixelCount:threshold:)`` is the weighted class pattern. +NNDSVD starts from an exact SVD where scikit-learn's is randomised. diff --git a/Sources/CuboVirtual/CuboVirtual.docc/CuboVirtual.md b/Sources/CuboVirtual/CuboVirtual.docc/CuboVirtual.md index 590ce3d..3320e75 100644 --- a/Sources/CuboVirtual/CuboVirtual.docc/CuboVirtual.md +++ b/Sources/CuboVirtual/CuboVirtual.docc/CuboVirtual.md @@ -3,6 +3,10 @@ Virtual detectors and the reductions over a cube: virtual images, the mean, median and maximum diffraction pattern, the centre of mass, probe size and origins. Each runs on the CPU or with Metal, behind one API. +``ResidentCube`` holds a cube in one Metal buffer for repeated reductions; ask ``ResidentCube/availability(of:physicalMemory:)`` +first, since it is capped at half of the physical memory. ``PolarCube`` is py4DSTEM's polar-elliptical transform of every +pattern and ``RadialBackground`` its radial background, subtraction and local average; both run on the CPU in Double. + ## Topics ### Articles @@ -35,9 +39,23 @@ the centre of mass, probe size and origins. Each runs on the CPU or with Metal, - ``SigmoidProfile`` - ``SyntheticProbe`` +### Resident cube + +- ``ResidentCube`` +- ``ResidentCubeError`` + +### Polar cube and radial background + +- ``PolarCube`` +- ``PolarSampling`` +- ``PolarPattern`` +- ``PolarCubeData`` +- ``RadialBackground`` + ### Origins - ``OriginMeasurement`` +- ``OriginEstimator`` - ``OriginMap`` - ``GaussianFilter`` - ``FriedelOrigin`` diff --git a/docs/embedding.md b/docs/embedding.md index 88c427f..adb1832 100644 --- a/docs/embedding.md +++ b/docs/embedding.md @@ -1,10 +1,23 @@ -# PCA and k-means (CuboEmbedding) +# PCA, k-means and featurization (CuboEmbedding) `PCA` and `KMeans` take row-major Double matrices (`rows × columns`) and need no scan shape, settings, progress or provenance. Streaming a cube into a feature matrix stays with the caller; for data too large for memory the kit takes the covariance (below). All public values are Sendable. Reference: scikit-learn 1.4.2 (NumPy 2.5.0, SciPy 1.18.0, Python 3.12.4); a different scikit-learn is not a reference. +## Featurization, NMF and Gaussian mixtures (0.9.0) + +`Featurization` is py4DSTEM 0.14.19's `process/classification/featurization.py` on a `FeatureMatrix` (one row per scan +position): `pca`, `nmf` (`_nmf_single` with its merge of correlated weight columns), `gmm` (`_gmm_single`), `minMaxScaled`, +`shiftedPositive`, `reduceFeatures`, `concatenate` and `classPatterns`. `NMF` and `GaussianMixture` are scikit-learn +1.9.1's, and `NumPyRandomState` is NumPy's legacy `RandomState`, bit identical, so seeded runs follow py4DSTEM's. Rows, +states and tolerances are in `docs/parity.md` (Featurization). + +Not ported: `Featurization.from_braggvectors` (it needs the Bragg types; CuboEmbedding does not depend on CuboBragg), `ICA`, +`RobustScaler`, `spatial_separation`, `consensus`, `add_features` and `delete_features`. The `nmf` and `gmm` branches of +`get_class_DPs` are buggy upstream and not copied. Upstream `NMF(init=None)` with `random_state=None` is not bit +repeatable (an unseeded randomised SVD); NNDSVD here uses an exact SVD. + ## Primary path: scikit-learn - `PCA.fit(_:rows:columns:components:)` = `PCA(n_components=k, svd_solver="full")`. The covariance (divisor rows − 1) is diff --git a/docs/open-items.md b/docs/open-items.md index a8b8796..9cc340f 100644 --- a/docs/open-items.md +++ b/docs/open-items.md @@ -30,5 +30,5 @@ Decisions to keep an eye on: - `PolarCube` keeps the whole cube in Double when asked for (`PolarCubeData`: scan x 180 x ~180 doubles per pattern); a lazy `CubeSource` adapter or a Float32 variant is for the consumer that needs it. The polar cube and radial background are CPU only (1024 patterns of 256 x 256 in 0.26 s); a Metal path would be bound by the host copy. - `CuboBragg` carries its own small coordinate-descent NMF (`ClassificationNumerics.coordinateDescentNMF`, custom start only) and a Jacobi least-squares SVD because it does not depend on `CuboEmbedding`; a `Package.swift` edge `CuboBragg -> CuboEmbedding` removes the copy. - `Featurization.from_braggvectors`, ICA, `RobustScaler`, `spatial_separation` and `consensus` are not ported. The GMM k-means start is a transcription of scikit-learn 1.4.2's Lloyd loop with the data centred as 1.9.1 does (every pin agrees on labels). Upstream `NMF(init=None)` with `random_state=None` is not bit repeatable (an unseeded randomised SVD). -- `fit_ellipse_1D` with default `fitradii` raises `TypeError` upstream; the kit's default is `(0, min(shape)/2)`. The consumer-check and DocC pages do not yet list the new public names. +- `fit_ellipse_1D` with default `fitradii` raises `TypeError` upstream; the kit's default is `(0, min(shape)/2)`. The DocC landing pages list the new public names; `tools/consumer-check` does not exercise them yet. - DiffractoKit follow-up: consume the polar cube (`PolarCube`, `PolarCubeData`) and `EllipseFit` for the ring and ellipse work; no code there yet.