Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
15 changes: 15 additions & 0 deletions Sources/CuboBragg/CuboBragg.docc/CuboBragg.md
Original file line number Diff line number Diff line change
Expand Up @@ -22,6 +22,11 @@ is the median fill of non-finite pixels, public.
``BraggDiskOptions/detectorAdapted(detectorRows:detectorColumns:probeRadius:base:)``.
``ProbeRingHint`` is the app's suggestion for a ring-structured probe's radius.

``BraggVectorClassification`` is py4DSTEM's `braggvectorclassification`: peaks labelled by the nearest reference vector,
an NMF of the label matrix, then split, merge, accept and reject steps, and the class images and class patterns.
``BraggVectors/virtualImage(detector:)`` sums the peaks inside a ``BraggVirtualDetector`` at each scan position
(py4DSTEM's `get_virtual_image`) into a ``BraggVirtualImage``.

## Topics

### Essentials
Expand Down Expand Up @@ -49,6 +54,16 @@ is the median fill of non-finite pixels, public.
- ``BraggNonFinitePixels``
- ``BraggPrecision``

### Classification

- ``BraggVectorClassification``

### Virtual images from Bragg vectors

- ``BraggVirtualImage``
- ``BraggVirtualDetector``
- ``BraggDetectorCenter``

### Probe

- ``ProbeRingHint``
Expand Down
31 changes: 31 additions & 0 deletions Sources/CuboCore/CuboCore.docc/CuboCore.md
Original file line number Diff line number Diff line change
Expand Up @@ -3,6 +3,11 @@
The cube model every other module reads: a stack of diffraction patterns, one per scan position, held in memory or
memory-mapped from a NumPy `.npy` file, read as Float32 and tiled along the scan.

``Calibration`` carries py4DSTEM's calibration of a cube (pixel sizes, units, origin, ellipse, QR rotation and flip, probe);
``CubeEditing`` is py4DSTEM's reshaping of a `DataCube` (scan binning and thinning, padding, axis swaps, detector resampling including the Fourier resample, the masked-pixel median filter) and
returns an ``EditedCube`` with the calibration that goes with it. ``EllipseFit`` is py4DSTEM's
`process/calibration/ellipse.py`: the algebraic 1D fit, the amorphous-ring fit and the degenerate-ellipse constraint.

## Topics

### Articles
Expand Down Expand Up @@ -35,6 +40,32 @@ memory-mapped from a NumPy `.npy` file, read as Float32 and tiled along the scan
- ``NPYFile``
- ``NPYHeader``

### Calibration

- ``Calibration``
- ``CalibrationError``
- ``CalibrationMeasurement``
- ``DetectorUnits``
- ``EllipseCalibration``
- ``ScanField``
- ``CoordinateGrid``

### Editing a cube

- ``CubeEditing``
- ``EditedCube``
- ``FourierResampleSpecification``
- ``FourierResampleOptions``
- ``FrameTransform``

### Ellipse fits

- ``EllipseFit``
- ``EllipseFitPlane``
- ``EllipseParameters``
- ``EllipseFitResult``
- ``EllipseFitError``

### Origin fits

- ``OriginFit``
Expand Down
22 changes: 21 additions & 1 deletion Sources/CuboEmbedding/CuboEmbedding.docc/CuboEmbedding.md
Original file line number Diff line number Diff line change
@@ -1,6 +1,7 @@
# ``CuboEmbedding``

CPU principal component analysis and k-means, with scikit-learn behaviour as the primary API.
CPU principal component analysis, k-means, NMF and Gaussian mixtures, with scikit-learn behaviour as the primary API,
and py4DSTEM's featurization on top of them.

Use ``PCA/fit(_:rows:columns:components:variant:)`` and ``KMeans/fit(_:rows:columns:initialCentroids:maxIterations:tolerance:isCancelled:)``
for scikit-learn 1.4.2 conventions. ``PCA/Variant/diffractionEmbeddingApp``, ``KMeans/fitSeeded(_:rows:columns:groups:seed:maxIterations:isCancelled:)``
Expand All @@ -12,6 +13,11 @@ For data too large to hold, ``PCACovarianceAccumulator`` streams the mean and co
signs from a second pass. ``DiffractionFeatures/binnedVector(pattern:rows:columns:binnedSize:implementation:)`` has an
Accelerate implementation, and ``KMeans`` takes an `isCancelled` closure.

``Featurization`` is py4DSTEM's `process/classification/featurization.py` on a ``FeatureMatrix`` (one row per scan
position): PCA, iterative NMF, Gaussian mixtures, scaling, feature reduction and class patterns. It rests on ``NMF`` and
``GaussianMixture`` (scikit-learn 1.9.1) and on ``NumPyRandomState``, NumPy's legacy `RandomState`, bit identical, so
seeded runs match py4DSTEM's. What is not ported is listed in <doc:Scope>.

## Topics

### Essentials
Expand All @@ -38,3 +44,17 @@ Accelerate implementation, and ``KMeans`` takes an `isCancelled` closure.

- ``DiffractionFeatures``
- ``DiffractionFeatures/Implementation``

### Featurization

- ``Featurization``
- ``FeatureMatrix``
- ``FeatureReduction``
- ``FeaturizationNMFResult``
- ``FeaturizationError``

### NMF and Gaussian mixtures

- ``NMF``
- ``GaussianMixture``
- ``NumPyRandomState``
13 changes: 10 additions & 3 deletions Sources/CuboEmbedding/CuboEmbedding.docc/Scope.md
Original file line number Diff line number Diff line change
@@ -1,10 +1,17 @@
# Scope

``PCA`` and ``KMeans`` operate on row-major Double matrices supplied by the caller. Feature extraction from a cube,
settings and progress are not part of this module (the covariance can be streamed with ``PCACovarianceAccumulator``); ``DiffractionFeatures`` only turns one pattern into one
vector.
``PCA``, ``KMeans``, ``NMF`` and ``GaussianMixture`` operate on row-major Double matrices supplied by the caller.
Feature extraction from a cube, settings and progress are not part of this module (the covariance can be streamed with
``PCACovarianceAccumulator``); ``DiffractionFeatures`` only turns one pattern into one vector, and ``Featurization`` starts
from a ``FeatureMatrix`` the caller builds.

Primary behaviour is pinned to scikit-learn 1.4.2: the covariance route to PCA with its `svd_flip` sign convention, and
Lloyd's algorithm from caller-supplied centroids. mac4DSTEM's Float32 PCA, seeded k-means++ and feature binning are
named options pinned to its original source. Not provided: whitening, other solvers, random or k-means++ seeding on the
scikit-learn path, sample weights, prediction of new rows. See `docs/embedding.md`.

``Featurization`` follows py4DSTEM 0.14.19 with scikit-learn 1.9.1 numbers. Not ported: `Featurization.from_braggvectors`
(it needs the Bragg types, and this module does not depend on CuboBragg), `ICA`, `RobustScaler`, `spatial_separation`,
`consensus`, `add_features` and `delete_features`. The `nmf` and `gmm` branches of `get_class_DPs` are buggy upstream and
are not copied; ``Featurization/classPatterns(weights:patterns:pixelCount:threshold:)`` is the weighted class pattern.
NNDSVD starts from an exact SVD where scikit-learn's is randomised.
18 changes: 18 additions & 0 deletions Sources/CuboVirtual/CuboVirtual.docc/CuboVirtual.md
Original file line number Diff line number Diff line change
Expand Up @@ -3,6 +3,10 @@
Virtual detectors and the reductions over a cube: virtual images, the mean, median and maximum diffraction pattern,
the centre of mass, probe size and origins. Each runs on the CPU or with Metal, behind one API.

``ResidentCube`` holds a cube in one Metal buffer for repeated reductions; ask ``ResidentCube/availability(of:physicalMemory:)``
first, since it is capped at half of the physical memory. ``PolarCube`` is py4DSTEM's polar-elliptical transform of every
pattern and ``RadialBackground`` its radial background, subtraction and local average; both run on the CPU in Double.

## Topics

### Articles
Expand Down Expand Up @@ -35,9 +39,23 @@ the centre of mass, probe size and origins. Each runs on the CPU or with Metal,
- ``SigmoidProfile``
- ``SyntheticProbe``

### Resident cube

- ``ResidentCube``
- ``ResidentCubeError``

### Polar cube and radial background

- ``PolarCube``
- ``PolarSampling``
- ``PolarPattern``
- ``PolarCubeData``
- ``RadialBackground``

### Origins

- ``OriginMeasurement``
- ``OriginEstimator``
- ``OriginMap``
- ``GaussianFilter``
- ``FriedelOrigin``
Expand Down
15 changes: 14 additions & 1 deletion docs/embedding.md
Original file line number Diff line number Diff line change
@@ -1,10 +1,23 @@
# PCA and k-means (CuboEmbedding)
# PCA, k-means and featurization (CuboEmbedding)

`PCA` and `KMeans` take row-major Double matrices (`rows × columns`) and need no scan shape, settings, progress or
provenance. Streaming a cube into a feature matrix stays with the caller; for data too large for memory the kit takes the
covariance (below). All public values are Sendable.
Reference: scikit-learn 1.4.2 (NumPy 2.5.0, SciPy 1.18.0, Python 3.12.4); a different scikit-learn is not a reference.

## Featurization, NMF and Gaussian mixtures (0.9.0)

`Featurization` is py4DSTEM 0.14.19's `process/classification/featurization.py` on a `FeatureMatrix` (one row per scan
position): `pca`, `nmf` (`_nmf_single` with its merge of correlated weight columns), `gmm` (`_gmm_single`), `minMaxScaled`,
`shiftedPositive`, `reduceFeatures`, `concatenate` and `classPatterns`. `NMF` and `GaussianMixture` are scikit-learn
1.9.1's, and `NumPyRandomState` is NumPy's legacy `RandomState`, bit identical, so seeded runs follow py4DSTEM's. Rows,
states and tolerances are in `docs/parity.md` (Featurization).

Not ported: `Featurization.from_braggvectors` (it needs the Bragg types; CuboEmbedding does not depend on CuboBragg), `ICA`,
`RobustScaler`, `spatial_separation`, `consensus`, `add_features` and `delete_features`. The `nmf` and `gmm` branches of
`get_class_DPs` are buggy upstream and not copied. Upstream `NMF(init=None)` with `random_state=None` is not bit
repeatable (an unseeded randomised SVD); NNDSVD here uses an exact SVD.

## Primary path: scikit-learn

- `PCA.fit(_:rows:columns:components:)` = `PCA(n_components=k, svd_solver="full")`. The covariance (divisor rows − 1) is
Expand Down
2 changes: 1 addition & 1 deletion docs/open-items.md
Original file line number Diff line number Diff line change
Expand Up @@ -30,5 +30,5 @@ Decisions to keep an eye on:
- `PolarCube` keeps the whole cube in Double when asked for (`PolarCubeData`: scan x 180 x ~180 doubles per pattern); a lazy `CubeSource` adapter or a Float32 variant is for the consumer that needs it. The polar cube and radial background are CPU only (1024 patterns of 256 x 256 in 0.26 s); a Metal path would be bound by the host copy.
- `CuboBragg` carries its own small coordinate-descent NMF (`ClassificationNumerics.coordinateDescentNMF`, custom start only) and a Jacobi least-squares SVD because it does not depend on `CuboEmbedding`; a `Package.swift` edge `CuboBragg -> CuboEmbedding` removes the copy.
- `Featurization.from_braggvectors`, ICA, `RobustScaler`, `spatial_separation` and `consensus` are not ported. The GMM k-means start is a transcription of scikit-learn 1.4.2's Lloyd loop with the data centred as 1.9.1 does (every pin agrees on labels). Upstream `NMF(init=None)` with `random_state=None` is not bit repeatable (an unseeded randomised SVD).
- `fit_ellipse_1D` with default `fitradii` raises `TypeError` upstream; the kit's default is `(0, min(shape)/2)`. The consumer-check and DocC pages do not yet list the new public names.
- `fit_ellipse_1D` with default `fitradii` raises `TypeError` upstream; the kit's default is `(0, min(shape)/2)`. The DocC landing pages list the new public names; `tools/consumer-check` does not exercise them yet.
- DiffractoKit follow-up: consume the polar cube (`PolarCube`, `PolarCubeData`) and `EllipseFit` for the ring and ellipse work; no code there yet.
Loading