diff --git a/Sources/CuboBragg/CuboBragg.docc/CuboBragg.md b/Sources/CuboBragg/CuboBragg.docc/CuboBragg.md index 1e131f9..24f9dd1 100644 --- a/Sources/CuboBragg/CuboBragg.docc/CuboBragg.md +++ b/Sources/CuboBragg/CuboBragg.docc/CuboBragg.md @@ -22,6 +22,11 @@ is the median fill of non-finite pixels, public. ``BraggDiskOptions/detectorAdapted(detectorRows:detectorColumns:probeRadius:base:)``. ``ProbeRingHint`` is the app's suggestion for a ring-structured probe's radius. +``BraggVectorClassification`` is py4DSTEM's `braggvectorclassification`: peaks labelled by the nearest reference vector, +an NMF of the label matrix, then split, merge, accept and reject steps, and the class images and class patterns. +``BraggVectors/virtualImage(detector:)`` sums the peaks inside a ``BraggVirtualDetector`` at each scan position +(py4DSTEM's `get_virtual_image`) into a ``BraggVirtualImage``. + ## Topics ### Essentials @@ -49,6 +54,16 @@ is the median fill of non-finite pixels, public. - ``BraggNonFinitePixels`` - ``BraggPrecision`` +### Classification + +- ``BraggVectorClassification`` + +### Virtual images from Bragg vectors + +- ``BraggVirtualImage`` +- ``BraggVirtualDetector`` +- ``BraggDetectorCenter`` + ### Probe - ``ProbeRingHint`` diff --git a/Sources/CuboCore/CuboCore.docc/CuboCore.md b/Sources/CuboCore/CuboCore.docc/CuboCore.md index b6e0e16..5f1a5c5 100644 --- a/Sources/CuboCore/CuboCore.docc/CuboCore.md +++ b/Sources/CuboCore/CuboCore.docc/CuboCore.md @@ -3,6 +3,11 @@ The cube model every other module reads: a stack of diffraction patterns, one per scan position, held in memory or memory-mapped from a NumPy `.npy` file, read as Float32 and tiled along the scan. +``Calibration`` carries py4DSTEM's calibration of a cube (pixel sizes, units, origin, ellipse, QR rotation and flip, probe); +``CubeEditing`` is py4DSTEM's reshaping of a `DataCube` (scan binning and thinning, padding, axis swaps, detector resampling including the Fourier resample, the masked-pixel median filter) and +returns an ``EditedCube`` with the calibration that goes with it. ``EllipseFit`` is py4DSTEM's +`process/calibration/ellipse.py`: the algebraic 1D fit, the amorphous-ring fit and the degenerate-ellipse constraint. + ## Topics ### Articles @@ -35,6 +40,32 @@ memory-mapped from a NumPy `.npy` file, read as Float32 and tiled along the scan - ``NPYFile`` - ``NPYHeader`` +### Calibration + +- ``Calibration`` +- ``CalibrationError`` +- ``CalibrationMeasurement`` +- ``DetectorUnits`` +- ``EllipseCalibration`` +- ``ScanField`` +- ``CoordinateGrid`` + +### Editing a cube + +- ``CubeEditing`` +- ``EditedCube`` +- ``FourierResampleSpecification`` +- ``FourierResampleOptions`` +- ``FrameTransform`` + +### Ellipse fits + +- ``EllipseFit`` +- ``EllipseFitPlane`` +- ``EllipseParameters`` +- ``EllipseFitResult`` +- ``EllipseFitError`` + ### Origin fits - ``OriginFit`` diff --git a/Sources/CuboEmbedding/CuboEmbedding.docc/CuboEmbedding.md b/Sources/CuboEmbedding/CuboEmbedding.docc/CuboEmbedding.md index 250e867..801c91f 100644 --- a/Sources/CuboEmbedding/CuboEmbedding.docc/CuboEmbedding.md +++ b/Sources/CuboEmbedding/CuboEmbedding.docc/CuboEmbedding.md @@ -1,6 +1,7 @@ # ``CuboEmbedding`` -CPU principal component analysis and k-means, with scikit-learn behaviour as the primary API. +CPU principal component analysis, k-means, NMF and Gaussian mixtures, with scikit-learn behaviour as the primary API, +and py4DSTEM's featurization on top of them. Use ``PCA/fit(_:rows:columns:components:variant:)`` and ``KMeans/fit(_:rows:columns:initialCentroids:maxIterations:tolerance:isCancelled:)`` for scikit-learn 1.4.2 conventions. ``PCA/Variant/diffractionEmbeddingApp``, ``KMeans/fitSeeded(_:rows:columns:groups:seed:maxIterations:isCancelled:)`` @@ -12,6 +13,11 @@ For data too large to hold, ``PCACovarianceAccumulator`` streams the mean and co signs from a second pass. ``DiffractionFeatures/binnedVector(pattern:rows:columns:binnedSize:implementation:)`` has an Accelerate implementation, and ``KMeans`` takes an `isCancelled` closure. +``Featurization`` is py4DSTEM's `process/classification/featurization.py` on a ``FeatureMatrix`` (one row per scan +position): PCA, iterative NMF, Gaussian mixtures, scaling, feature reduction and class patterns. It rests on ``NMF`` and +``GaussianMixture`` (scikit-learn 1.9.1) and on ``NumPyRandomState``, NumPy's legacy `RandomState`, bit identical, so +seeded runs match py4DSTEM's. What is not ported is listed in . + ## Topics ### Essentials @@ -38,3 +44,17 @@ Accelerate implementation, and ``KMeans`` takes an `isCancelled` closure. - ``DiffractionFeatures`` - ``DiffractionFeatures/Implementation`` + +### Featurization + +- ``Featurization`` +- ``FeatureMatrix`` +- ``FeatureReduction`` +- ``FeaturizationNMFResult`` +- ``FeaturizationError`` + +### NMF and Gaussian mixtures + +- ``NMF`` +- ``GaussianMixture`` +- ``NumPyRandomState`` diff --git a/Sources/CuboEmbedding/CuboEmbedding.docc/Scope.md b/Sources/CuboEmbedding/CuboEmbedding.docc/Scope.md index 2764c00..4e9bf51 100644 --- a/Sources/CuboEmbedding/CuboEmbedding.docc/Scope.md +++ b/Sources/CuboEmbedding/CuboEmbedding.docc/Scope.md @@ -1,10 +1,17 @@ # Scope -``PCA`` and ``KMeans`` operate on row-major Double matrices supplied by the caller. Feature extraction from a cube, -settings and progress are not part of this module (the covariance can be streamed with ``PCACovarianceAccumulator``); ``DiffractionFeatures`` only turns one pattern into one -vector. +``PCA``, ``KMeans``, ``NMF`` and ``GaussianMixture`` operate on row-major Double matrices supplied by the caller. +Feature extraction from a cube, settings and progress are not part of this module (the covariance can be streamed with +``PCACovarianceAccumulator``); ``DiffractionFeatures`` only turns one pattern into one vector, and ``Featurization`` starts +from a ``FeatureMatrix`` the caller builds. Primary behaviour is pinned to scikit-learn 1.4.2: the covariance route to PCA with its `svd_flip` sign convention, and Lloyd's algorithm from caller-supplied centroids. mac4DSTEM's Float32 PCA, seeded k-means++ and feature binning are named options pinned to its original source. Not provided: whitening, other solvers, random or k-means++ seeding on the scikit-learn path, sample weights, prediction of new rows. See `docs/embedding.md`. + +``Featurization`` follows py4DSTEM 0.14.19 with scikit-learn 1.9.1 numbers. Not ported: `Featurization.from_braggvectors` +(it needs the Bragg types, and this module does not depend on CuboBragg), `ICA`, `RobustScaler`, `spatial_separation`, +`consensus`, `add_features` and `delete_features`. The `nmf` and `gmm` branches of `get_class_DPs` are buggy upstream and +are not copied; ``Featurization/classPatterns(weights:patterns:pixelCount:threshold:)`` is the weighted class pattern. +NNDSVD starts from an exact SVD where scikit-learn's is randomised. diff --git a/Sources/CuboVirtual/CuboVirtual.docc/CuboVirtual.md b/Sources/CuboVirtual/CuboVirtual.docc/CuboVirtual.md index 590ce3d..3320e75 100644 --- a/Sources/CuboVirtual/CuboVirtual.docc/CuboVirtual.md +++ b/Sources/CuboVirtual/CuboVirtual.docc/CuboVirtual.md @@ -3,6 +3,10 @@ Virtual detectors and the reductions over a cube: virtual images, the mean, median and maximum diffraction pattern, the centre of mass, probe size and origins. Each runs on the CPU or with Metal, behind one API. +``ResidentCube`` holds a cube in one Metal buffer for repeated reductions; ask ``ResidentCube/availability(of:physicalMemory:)`` +first, since it is capped at half of the physical memory. ``PolarCube`` is py4DSTEM's polar-elliptical transform of every +pattern and ``RadialBackground`` its radial background, subtraction and local average; both run on the CPU in Double. + ## Topics ### Articles @@ -35,9 +39,23 @@ the centre of mass, probe size and origins. Each runs on the CPU or with Metal, - ``SigmoidProfile`` - ``SyntheticProbe`` +### Resident cube + +- ``ResidentCube`` +- ``ResidentCubeError`` + +### Polar cube and radial background + +- ``PolarCube`` +- ``PolarSampling`` +- ``PolarPattern`` +- ``PolarCubeData`` +- ``RadialBackground`` + ### Origins - ``OriginMeasurement`` +- ``OriginEstimator`` - ``OriginMap`` - ``GaussianFilter`` - ``FriedelOrigin`` diff --git a/docs/embedding.md b/docs/embedding.md index 88c427f..adb1832 100644 --- a/docs/embedding.md +++ b/docs/embedding.md @@ -1,10 +1,23 @@ -# PCA and k-means (CuboEmbedding) +# PCA, k-means and featurization (CuboEmbedding) `PCA` and `KMeans` take row-major Double matrices (`rows × columns`) and need no scan shape, settings, progress or provenance. Streaming a cube into a feature matrix stays with the caller; for data too large for memory the kit takes the covariance (below). All public values are Sendable. Reference: scikit-learn 1.4.2 (NumPy 2.5.0, SciPy 1.18.0, Python 3.12.4); a different scikit-learn is not a reference. +## Featurization, NMF and Gaussian mixtures (0.9.0) + +`Featurization` is py4DSTEM 0.14.19's `process/classification/featurization.py` on a `FeatureMatrix` (one row per scan +position): `pca`, `nmf` (`_nmf_single` with its merge of correlated weight columns), `gmm` (`_gmm_single`), `minMaxScaled`, +`shiftedPositive`, `reduceFeatures`, `concatenate` and `classPatterns`. `NMF` and `GaussianMixture` are scikit-learn +1.9.1's, and `NumPyRandomState` is NumPy's legacy `RandomState`, bit identical, so seeded runs follow py4DSTEM's. Rows, +states and tolerances are in `docs/parity.md` (Featurization). + +Not ported: `Featurization.from_braggvectors` (it needs the Bragg types; CuboEmbedding does not depend on CuboBragg), `ICA`, +`RobustScaler`, `spatial_separation`, `consensus`, `add_features` and `delete_features`. The `nmf` and `gmm` branches of +`get_class_DPs` are buggy upstream and not copied. Upstream `NMF(init=None)` with `random_state=None` is not bit +repeatable (an unseeded randomised SVD); NNDSVD here uses an exact SVD. + ## Primary path: scikit-learn - `PCA.fit(_:rows:columns:components:)` = `PCA(n_components=k, svd_solver="full")`. The covariance (divisor rows − 1) is diff --git a/docs/open-items.md b/docs/open-items.md index a8b8796..9cc340f 100644 --- a/docs/open-items.md +++ b/docs/open-items.md @@ -30,5 +30,5 @@ Decisions to keep an eye on: - `PolarCube` keeps the whole cube in Double when asked for (`PolarCubeData`: scan x 180 x ~180 doubles per pattern); a lazy `CubeSource` adapter or a Float32 variant is for the consumer that needs it. The polar cube and radial background are CPU only (1024 patterns of 256 x 256 in 0.26 s); a Metal path would be bound by the host copy. - `CuboBragg` carries its own small coordinate-descent NMF (`ClassificationNumerics.coordinateDescentNMF`, custom start only) and a Jacobi least-squares SVD because it does not depend on `CuboEmbedding`; a `Package.swift` edge `CuboBragg -> CuboEmbedding` removes the copy. - `Featurization.from_braggvectors`, ICA, `RobustScaler`, `spatial_separation` and `consensus` are not ported. The GMM k-means start is a transcription of scikit-learn 1.4.2's Lloyd loop with the data centred as 1.9.1 does (every pin agrees on labels). Upstream `NMF(init=None)` with `random_state=None` is not bit repeatable (an unseeded randomised SVD). -- `fit_ellipse_1D` with default `fitradii` raises `TypeError` upstream; the kit's default is `(0, min(shape)/2)`. The consumer-check and DocC pages do not yet list the new public names. +- `fit_ellipse_1D` with default `fitradii` raises `TypeError` upstream; the kit's default is `(0, min(shape)/2)`. The DocC landing pages list the new public names; `tools/consumer-check` does not exercise them yet. - DiffractoKit follow-up: consume the polar cube (`PolarCube`, `PolarCubeData`) and `EllipseFit` for the ring and ellipse work; no code there yet.