diff --git a/docs/architecture.md b/docs/architecture.md index 63085c4..2515c92 100644 --- a/docs/architecture.md +++ b/docs/architecture.md @@ -50,6 +50,22 @@ commands/.py orchestration: selects adapter + backend, drives the stage
+## Checkpointing + +`stamp.utils.checkpoint` provides shared checkpointing/resume utilities: `sync_checkpoint(work_dir, fingerprint)` keeps a work directory when its stored `checkpoint.json` matches the current fingerprint (normalised through JSON) and wipes it otherwise; `signatures` and `directory_signatures` fingerprint input files by size and modification time; `atomic_save_npz` and `atomic_write_json` write each finished unit so a partial file never looks complete. + +Command | Work directory | Unit checkpointed +-- | -- | -- +`pick` | `raw//` | tomogram picks (`.json`) +`decoy` | `raw//` | rejected-surface: surface scores (`scores/.npz`); shifted: surfaces (`surfaces/.npz`); synthetic-noise: volumes and picks +`classify` | `raw/` | subvolumes, per-tomogram features, per-cluster alignment angles +`identify` | `raw/{real,decoy}/` | one fit score per (class, candidate) +`refine` | `//` | finished half-maps + +Random sampling steps (decoy placement) are never checkpointed, only the deterministic work before them, so a resumed run is identical to an uninterrupted one for a given seed. + +
+ ## Adapters & Runners STAMP separates what a tool is from how to run it, as a `ToolAdapter` and `Runner` respectively: - `ToolAdapter`: builds a `ToolCommand` from `AdapterInputs` and parses a `RunResult` into `AdapterOutput`. diff --git a/docs/cli.md b/docs/cli.md index 0fb4097..0f18c95 100644 --- a/docs/cli.md +++ b/docs/cli.md @@ -38,6 +38,16 @@ Option | Default | Meaning | Notes
+## Resuming interrupted commands + +STAMP's `pick`, `decoy`, `classify`, `identify` and `refine` commands periodically checkpoint finished work throughout processing. Re-running a command with the same inputs/parameters will be detected, and the command will resume from the previous checkpoint. Partial output lives in `raw/` (for `refine`, in the class directory) alongside a `checkpoint.json` fingerprint of everything that affects the results. + +If a command's inputs or parameters differ, any old partial output is discarded with a warning. Parameters which don't affect results (e.g. `--n-processes`, plots, `--keep-raw`) and steps applied after checkpointed work (for `pick`: consensus rule, distance threshold, seed and beam angle filter) can change between runs without invalidating a given checkpoint. + +`stamp run` re-runs will also reuse any valid partial output within a stage. + +
+ ## `stamp pick` ``` @@ -64,6 +74,8 @@ Argument/Option | Default | Use `--max-beam-angle-deviation` | `None` | Drop particles whose normal deviates from the beam-orthogonal plane by more than this many degrees `--vesicle-labels-mrc` | `None` | Path to an MRC file produced by EValuator's `label` command, or a directory to be matched by filename stem `--normalise-per-vesicle` | `False` | Normalise `stamp-native` scores per vesicle instead of per tomogram +`-n, --n-processes` | `1` | Processes to use for the `stamp-native` picker (1 = sequential) +`--keep-raw` | `False` | Keep the raw intermediate output instead of archiving it to `raw.tar.gz`
@@ -82,20 +94,22 @@ Option | Default | Use `-s, --seg-dir` | *n/a (required unless `--method synthetic-noise`)* | Directory containing segmented MRC files `-r, --raw-dir` | *n/a (required unless `--method synthetic-noise`)* | Directory of raw tomogram MRCs `--picker-params` | `'{}'` | JSON of native-picker parameters (must match the values used for `stamp pick` so decoys are drawn from the same candidate pool) -`--n-decoys-per-tomogram` | `50` | Number of decoy positions to pick per tomogram -`--min-distance-from-real-a` | `100.0` | Minimum distance from a real pick in Å for `--method rejected-surface` +`--n-decoys-per-tomogram` | `None` *(adaptive: matches the real pick count per tomogram)* | Number of decoy positions to pick per tomogram (required for `--method synthetic-noise`) +`--min-distance-from-real-a` | `100.0` | Minimum distance in Å from a real pick for `--method rejected-surface`, or from the membrane surface for `--method shifted` `--min-pick-distance-a` | `60.0` | Minimum distance from a real pick in Å for `--method shifted` `--min-shift-a` | `200.0` | Minimum displacement in Å for `--method shifted` `--max-shift-a` | `600.0` | Maximum displacement in Å for `--method shifted` `--n-synthetic-tomograms` | `3` | Number of volumes to produce for `--method synthetic-noise` `--synthetic-shape` | `'200,200,200'` | Volume shape (in `z,y,x`) to use for `--method synthetic-noise` `--seed` | `0` | Seed to use for sampling and half-set assignment -`-backend` | `local` | Backend to use for picking (`local` or `mock`) +`--backend` | `local` | Backend to use for picking (`local` or `mock`) `--plots/--no-plots` | `--plots` | Write decoy position plots `--pick-plot-style` | `segmented` | Style of decoy position plots (`segmented` or `none`) `--plot-format` | `tiff` | Decoy plot file format (`png` or `jpg` or `tiff` or `svg`) `--pick-zstack-movie/--no-pick-zstack-movie` | `--pick-zstack-movie` | Write a per-tomogram movie through Z with picks highlighted `--pick-plot-3d/--no-pick-plot-3d` | `--no-pick-plot-3d` | Write a static 3D pick visualisation per tomogram +`-n, --n-processes` | `1` | Processes to use for per-tomogram decoy generation (1 = sequential) +`--keep-raw` | `False` | Keep the raw intermediate output instead of archiving it to `raw.tar.gz` ### `stamp decoy` methods `stamp decoy` provides three methods for generating decoy picks: @@ -137,6 +151,8 @@ Option | Default | Use `--n-azimuthal-samples` | `64` | Number of azimuthal sampling points (must be >= `2*(modes+1)`) `--plots/--no-plots` | `--plots` | Write embedding and class-average plots `--plot-format` | `tiff` | Embedding and class average plot file format (`png` or `jpg` or `tiff` or `svg`) +`-n, --n-processes` | `1` | Processes to use for per-cluster in-plane alignment, extraction and feature building (1 = sequential) +`--keep-raw` | `False` | Keep the raw intermediate output instead of archiving it to `raw.tar.gz`
@@ -159,6 +175,7 @@ Option | Default | Use `--fetch-missing` | `False` | Fetch model from AlphaFold when a candidate has no local `structure_path` `--plots/--no-plots` | `--plots` | Write fit-score heatmap and decoy-control histogram plots `--plot-format` | `tiff` | Fit-score heatmap and decoy-control histogram plot file format (`png` or `jpg` or `tiff` or `svg`) +`-n, --n-processes` | `1` | Processes to use for candidate fitting (1 = sequential)
@@ -181,7 +198,7 @@ Option | Default | Use `--mask` | `None` | Path to MRC file to use as mask `--iterations` | `5` | Number of refinement iterations to use `--backend` | `local` | Backend to use for refinement (`local` or `mock`) -`--combined-halfset` | `False` | Use a single combined refinement step with an internal split, instead of two independent trees +`--threshold` | `0.143` | FSC threshold used to read the resolution `--plots/--no-plots` | `--plots` | Write FSC curve plots `--plot-format` | `tiff` | FSC curve plot file format (`png` or `jpg` or `tiff` or `svg`) diff --git a/pyproject.toml b/pyproject.toml index 00a0f39..a52da59 100644 --- a/pyproject.toml +++ b/pyproject.toml @@ -1,6 +1,6 @@ [project] name = "stamp" -version = "0.12.0" +version = "0.13.0" description = "A workflow for Sub-Tomogram Averaging Membrane Proteins" readme = "README.md" authors = [ @@ -8,6 +8,7 @@ authors = [ ] requires-python = ">=3.14" dependencies = [ + "fast-hdbscan>=0.3.2", "loguru>=0.7.3", "matplotlib>=3.11.1", "mrcfile>=1.5.4", diff --git a/src/stamp/adapters/native.py b/src/stamp/adapters/native.py index af3d017..410475e 100644 --- a/src/stamp/adapters/native.py +++ b/src/stamp/adapters/native.py @@ -90,5 +90,9 @@ def _on_error(job: _PickJob, exc: Exception) -> None: log.error(f'{job.tomogram_id}: {exc}') else: raise exc - run_parallel(jobs, _pick_one, max_workers=n_workers, label='stamp-native', on_success=_on_success, on_error=_on_error) + # a tomogram's cache file is written atomically on completion, so its presence means it is done + pending = [job for job in jobs if not (inputs.output_directory / f'{job.tomogram_id}.json').is_file()] + if len(pending) < len(jobs): + log.info(f'Resuming stamp-native: {len(jobs) - len(pending)} tomogram(s) loaded from checkpoint') + run_parallel(pending, _pick_one, max_workers=n_workers, label='stamp-native', on_success=_on_success, on_error=_on_error) return load_cached_picks(inputs.output_directory, inputs.tomogram_ids) diff --git a/src/stamp/classify/cluster.py b/src/stamp/classify/cluster.py index 23d1542..990bef3 100644 --- a/src/stamp/classify/cluster.py +++ b/src/stamp/classify/cluster.py @@ -5,11 +5,11 @@ # Import external dependencies import numpy as np from dataclasses import dataclass -from sklearn.cluster import HDBSCAN, KMeans +from sklearn.cluster import KMeans from sklearn.decomposition import PCA # Import internal STAMP objects -from stamp.utils.log import log +from stamp.utils.log import log, log_config # NOISE_CLUSTER_ID: id for noise cluster NOISE_CLUSTER_ID = 'noise' @@ -40,48 +40,51 @@ def _cluster_embedding(embedding: np.ndarray, config: ClusteringConfig) -> tuple if config.method == 'hdbscan': if embedding.shape[0] < config.min_cluster_size: raise ValueError(f'only {embedding.shape[0]} particles but min_cluster_size is {config.min_cluster_size}') - labels = HDBSCAN(min_cluster_size=config.min_cluster_size).fit_predict(embedding) + from fast_hdbscan import HDBSCAN + labels = HDBSCAN(min_cluster_size=config.min_cluster_size, min_samples=max(config.min_cluster_size - 1, 1)).fit_predict(np.asarray(embedding, dtype=np.float64)) else: n_clusters = min(config.n_clusters, embedding.shape[0]) labels = KMeans(n_clusters=n_clusters, random_state=config.random_state, n_init=10).fit_predict(embedding) centroids = {int(label): embedding[labels == label].mean(axis=0) for label in np.unique(labels) if label != -1} return labels, centroids -# reduce_and_cluster: PCA followed by HDBSCAN or KMeans -def reduce_and_cluster(features: np.ndarray, config: ClusteringConfig) -> ClusteringResult: +# reduce_features: PCA of a feature matrix, return (embedding, explained variance ratio) +def reduce_features(features: np.ndarray, config: ClusteringConfig) -> tuple[np.ndarray, np.ndarray]: if features.shape[0] == 0: raise ValueError('cannot cluster an empty feature matrix') n_components = min(config.n_components, features.shape[0] - 1, features.shape[1]) if n_components < 2: raise ValueError(f'Too few particles ({features.shape[0]}) to run PCA (at least 2 needed)') - + log.progress(f'Starting PCA: {features.shape[0]} particles, {features.shape[1]} features, {n_components} components') pca = PCA(n_components=n_components, random_state=config.random_state) embedding = pca.fit_transform(features) - log.debug(f'PCA: {n_components} components, {pca.explained_variance_ratio_[:n_components].sum():.1%} variance explained') + log.progress(f'PCA complete: {pca.explained_variance_ratio_.sum():.1%} variance explained') + return embedding, pca.explained_variance_ratio_ + + +# reduce_and_cluster: PCA followed by HDBSCAN or KMeans +def reduce_and_cluster(features: np.ndarray, config: ClusteringConfig) -> ClusteringResult: + embedding, explained_variance_ratio = reduce_features(features, config) + log.progress(f'Starting {config.method} clustering') + log_config(config.method, config) labels, centroids = _cluster_embedding(embedding, config) - log.debug(f'{config.method}: {len(set(labels))} cluster(s) found') - return ClusteringResult( - labels=labels, - embedding=embedding, - explained_variance_ratio=pca.explained_variance_ratio_, - centroids=centroids, - ) + log.progress(f'Clustering complete: {len(set(labels) - {-1})} cluster(s), {int((labels == -1).sum())} noise') + return ClusteringResult(labels=labels, embedding=embedding, explained_variance_ratio=explained_variance_ratio, centroids=centroids) # reduce_and_cluster_shared: one PCA basis fitted on every particle, each index group clustered in that basis def reduce_and_cluster_shared(features: np.ndarray, groups: dict[str, np.ndarray], config: ClusteringConfig) -> dict[str, ClusteringResult]: - n_components = min(config.n_components, features.shape[0] - 1, features.shape[1]) - if n_components < 1: - raise ValueError(f'too few particles ({features.shape[0]}) to run PCA (at least 2 needed)') - pca = PCA(n_components=n_components, random_state=config.random_state) - embedding_all = pca.fit_transform(features) + embedding_all, explained_variance_ratio = reduce_features(features, config) results: dict[str, ClusteringResult] = {} for name, indices in groups.items(): + log.progress(f'Starting {config.method} clustering for half {name}') + log_config(config.method, config) embedding = embedding_all[indices] labels, centroids = _cluster_embedding(embedding, config) + log.progress(f'Half {name} clustering complete: {len(set(labels) - {-1})} cluster(s), {int((labels == -1).sum())} noise') results[name] = ClusteringResult( labels=labels, embedding=embedding, - explained_variance_ratio=pca.explained_variance_ratio_, + explained_variance_ratio=explained_variance_ratio, centroids=centroids, ) return results diff --git a/src/stamp/classify/features.py b/src/stamp/classify/features.py index 75f7e01..c56759b 100644 --- a/src/stamp/classify/features.py +++ b/src/stamp/classify/features.py @@ -151,6 +151,7 @@ def build_feature_matrix( by_tomogram.setdefault(tomogram_id, []).append(local_index) with pool_context as pool: completed = 0 + n_resumed = 0 for tomogram_id, local_indices in by_tomogram.items(): rows = [subvols.index[i][1] for i in local_indices] tomogram_magnitudes = _load_cached_magnitudes(cache_dir, tomogram_id, rows) @@ -170,10 +171,13 @@ def build_feature_matrix( if cache_dir is not None: atomic_save_npz(cache_dir / f'{tomogram_id}.npz', rows=np.array(rows), magnitudes=tomogram_magnitudes) else: + n_resumed += 1 log.debug(f'{tomogram_id}: features loaded from checkpoint') completed += len(local_indices) for local_index, result in zip(local_indices, tomogram_magnitudes): magnitudes[local_index] = result + if n_resumed: + log.info(f'Resuming feature extraction: {n_resumed}/{len(by_tomogram)} tomogram(s) loaded from checkpoint') stacked = np.stack(magnitudes) # (n_particles, n_modes, n_radial_bins, box_voxels) n_particles = stacked.shape[0] diff --git a/src/stamp/commands/classify.py b/src/stamp/commands/classify.py index f6a01dc..cff1184 100644 --- a/src/stamp/commands/classify.py +++ b/src/stamp/commands/classify.py @@ -3,7 +3,7 @@ ''' # Import external dependencies -import json, matplotlib.pyplot as plt, numpy as np +import json, os, matplotlib.pyplot as plt, numpy as np from pathlib import Path # Import internal STAMP objects @@ -17,7 +17,7 @@ reduce_and_cluster, reduce_and_cluster_shared, ) -from stamp.utils.checkpoint import file_signature, sync_checkpoint +from stamp.utils.checkpoint import file_signature, signatures, sync_checkpoint from stamp.classify.extract import extract_particle_set from stamp.classify.features import build_feature_matrix from stamp.backends.base import ToolCommand @@ -84,8 +84,8 @@ def run_classify( # everything that changes results; n_workers, plots and keep_raw are left out so a run can resume with different resources fingerprint = { 'particles': file_signature(particles, content=True), - 'raw': {stem: file_signature(Path(path)) for stem, path in sorted(tomogram_paths.items())}, - 'segmentation': {stem: file_signature(Path(path)) for stem, path in sorted(segmentation_paths.items())}, + 'raw': signatures(tomogram_paths), + 'segmentation': signatures(segmentation_paths), 'voxel_size_angstrom': voxel_size_angstrom, 'box_voxels': box_voxels, 'n_radial_bins': n_radial_bins, @@ -125,7 +125,7 @@ def run_classify( n_workers=n_workers, cache_dir=output_dir / 'raw' / 'features', ) - log.debug(f'Feature vector: {features.shape[1]} dimensions (modes 0-{azimuthal_modes})') + log.progress(f'Feature matrix complete: {features.shape[0]} particles x {features.shape[1]} features (modes 0-{azimuthal_modes})') degenerate = ~features.any(axis=1) if degenerate.any(): @@ -145,6 +145,9 @@ def run_classify( random_state=random_state, ) log_config('classify.clustering', config) + + # set environment variable NUMBA_NUM_THREADS so fast_hdbscan doesn't try to use all CPUs on compute clusters + os.environ['NUMBA_NUM_THREADS'] = str(n_workers) output_dir.mkdir(parents=True, exist_ok=True) diff --git a/src/stamp/commands/decoy.py b/src/stamp/commands/decoy.py index 2323b3e..5a60570 100644 --- a/src/stamp/commands/decoy.py +++ b/src/stamp/commands/decoy.py @@ -4,6 +4,7 @@ # Import external dependencies import json +from dataclasses import asdict from pathlib import Path # Import internal STAMP objects @@ -20,6 +21,7 @@ from stamp.picking.native import NativePickerConfig from stamp.run.state import stage_dir from stamp.schemas.particles import ParticleSet +from stamp.utils.checkpoint import directory_signatures, file_signature, sync_checkpoint from stamp.utils.errors import StampPipelineError, StampValidationError from stamp.utils.io import archive_and_remove_directory, load_tomogram_manifests, resolve_directory_voxel_size_angstrom, write_sidecar from stamp.utils.log import log @@ -67,6 +69,26 @@ def run_decoy( output_dir.mkdir(parents=True, exist_ok=True) + # everything that changes decoy positions; workers, plots and keep_raw are left out so a run can resume with different resources + uses_real_data = method != METHOD_SYNTHETIC_NOISE + fingerprint = { + 'method': method, + 'seed': seed, + 'picker_config': asdict(config), + 'n_decoys_per_tomogram': n_decoys_per_tomogram, + 'min_distance_from_real_angstrom': min_distance_from_real_angstrom, + 'min_shift_angstrom': min_shift_angstrom, + 'max_shift_angstrom': max_shift_angstrom, + 'min_distance_from_picks_angstrom': min_distance_from_picks_angstrom, + 'n_synthetic_tomograms': n_synthetic_tomograms, + 'synthetic_shape': synthetic_shape, + 'real_particle_set': file_signature(real_particle_set, content=True) if uses_real_data and real_particle_set else None, + 'segmentation': directory_signatures(segmentation_dir) if uses_real_data else {}, + 'raw': directory_signatures(raw_tomogram_dir) if uses_real_data else {}, + } + if sync_checkpoint(output_dir / 'raw', fingerprint): + log.info('Found matching partial output, resuming') + if method == METHOD_SYNTHETIC_NOISE: shape = tuple(int(value) for value in synthetic_shape.split(',')) if len(shape) != 3: diff --git a/src/stamp/commands/identify.py b/src/stamp/commands/identify.py index 03169b3..faa14c6 100644 --- a/src/stamp/commands/identify.py +++ b/src/stamp/commands/identify.py @@ -3,7 +3,7 @@ ''' # Import external dependencies -import json, matplotlib.pyplot as plt, mrcfile, numpy as np, re, tomllib +import json, matplotlib.pyplot as plt, mrcfile, numpy as np, re, shutil, tomllib from collections import defaultdict from dataclasses import dataclass from pathlib import Path @@ -19,6 +19,7 @@ from stamp.identify.fit import fit_candidate, rank_candidates from stamp.identify.panel import Candidate, load_candidate_panel from stamp.identify.simulate import simulate_density, to_comparable +from stamp.utils.checkpoint import atomic_write_json, directory_signatures, signatures, sync_checkpoint from stamp.utils.errors import StampPipelineError from stamp.utils.io import write_sidecar from stamp.utils.log import log @@ -68,19 +69,43 @@ def _fit_one(job: _FitJob) -> tuple[str, str, float]: score = fit_candidate(job.comparable_average, simulated) return job.class_id, job.candidate.name, score -# _score_panel: fit every candidate to every class average, returns {class_id: {candidate: score}} -def _score_panel(class_averages, panel, resolution, fitter, backend, n_workers: int = 1): - jobs = [ - _FitJob(class_id, to_comparable(average, voxel_size, resolution), average.shape[-1], voxel_size, resolution, candidate) - for class_id, (average, voxel_size) in class_averages.items() - for candidate in panel - ] +# _score_path: checkpoint file for one (class, candidate) score; the name is sanitised for the filesystem, the payload keeps the true names +def _score_path(checkpoint_dir: Path, class_id: str, candidate_name: str) -> Path: + return checkpoint_dir / f'{class_id}__{re.sub(r"[^\w.-]", "_", candidate_name)}.json' + +# _load_score: a checkpointed score, or None when absent, unreadable or for a different (class, candidate) +def _load_score(checkpoint_dir: Path | None, class_id: str, candidate_name: str) -> float | None: + if checkpoint_dir is None: + return None + try: + saved = json.loads(_score_path(checkpoint_dir, class_id, candidate_name).read_text()) + if saved['class_id'] != class_id or saved['candidate'] != candidate_name: + return None + return float(saved['score']) + except (OSError, json.JSONDecodeError, KeyError, TypeError, ValueError): + return None + +# _score_panel: fit every candidate to every class average, each finished fit checkpointed in checkpoint_dir when given, returns {class_id: {candidate: score}} +def _score_panel(class_averages, panel, resolution, fitter, backend, n_workers: int = 1, checkpoint_dir: Path | None = None): all_scores: dict[str, dict[str, float]] = {class_id: {} for class_id in class_averages} + jobs = [] + for class_id, (average, voxel_size) in class_averages.items(): + for candidate in panel: + saved = _load_score(checkpoint_dir, class_id, candidate.name) + if saved is None: + jobs.append(_FitJob(class_id, to_comparable(average, voxel_size, resolution), average.shape[-1], voxel_size, resolution, candidate)) + else: + all_scores[class_id][candidate.name] = saved + n_resumed = sum(len(scores) for scores in all_scores.values()) + if n_resumed: + log.info(f'Resuming identification: {n_resumed} score(s) loaded from checkpoint') def _on_success(job: _FitJob, result: tuple[str, str, float]) -> None: class_id, candidate_name, score = result all_scores[class_id][candidate_name] = score log.debug(f'{class_id}/{candidate_name}: score={score:.3f}') + if checkpoint_dir is not None: + atomic_write_json(_score_path(checkpoint_dir, class_id, candidate_name), {'class_id': class_id, 'candidate': candidate_name, 'score': float(score)}) run_parallel(jobs, _fit_one, max_workers=n_workers, label='identify-fit', on_success=_on_success) return all_scores @@ -108,8 +133,20 @@ def run_identify( log.info(f'Loaded {len(class_averages)} class averages') inplane_aligned = _classify_was_inplane_aligned(classes) + # everything that changes scores; n_workers, backend and plots are left out so a run can resume with different resources + fingerprint = { + 'classes': directory_signatures(classes), + 'decoy_classes': directory_signatures(decoy_classes), + 'candidates': signatures({'panel': candidates, **{f'structure:{candidate.name}': candidate.structure_path for candidate in panel}}), + 'resolution': resolution, + 'fitter': fitter, + } + checkpoint_dir = output_dir / 'raw' + if sync_checkpoint(checkpoint_dir, fingerprint): + log.info('Found matching partial output, resuming') + log.progress(f'Fitting {len(panel)} candidates against {len(class_averages)} classes') - real_scores = _score_panel(class_averages, panel, resolution, fitter, backend, n_workers) + real_scores = _score_panel(class_averages, panel, resolution, fitter, backend, n_workers, checkpoint_dir / 'real') results = [rank_candidates(class_id, scores, method=f'stamp-{fitter}') for class_id, scores in sorted(real_scores.items())] output_dir.mkdir(parents=True, exist_ok=True) @@ -119,7 +156,7 @@ def run_identify( real_best, decoy_best = None, None if decoy_classes is not None: log.progress('Fitting candidates against decoy classes for control') - decoy_scores = _score_panel(load_class_averages(decoy_classes), panel, resolution, fitter, backend, n_workers) + decoy_scores = _score_panel(load_class_averages(decoy_classes), panel, resolution, fitter, backend, n_workers, checkpoint_dir / 'decoy') real_best = [max(s.values()) for s in real_scores.values()] decoy_best = [max(s.values()) for s in decoy_scores.values()] decoy_control = evaluate_decoy_control(real_best, decoy_best) @@ -153,6 +190,7 @@ def run_identify( + ([('decoy_classes', decoy_classes)] if decoy_classes else []), ) log.info(f'Wrote identification for {len(results)} classes to {output_dir}') + shutil.rmtree(checkpoint_dir, ignore_errors=True) # identify has no keep_raw, the checkpoint is only needed until it completes if make_plots: _plot_identify(real_scores, results, decoy_control, real_best, decoy_best, output_dir, plot_format) diff --git a/src/stamp/commands/pick.py b/src/stamp/commands/pick.py index 22c90ee..1774a67 100644 --- a/src/stamp/commands/pick.py +++ b/src/stamp/commands/pick.py @@ -21,6 +21,7 @@ from stamp.run.state import stage_dir from stamp.schemas.manifest import TomogramManifest from stamp.schemas.picks import RawPick +from stamp.utils.checkpoint import signatures, sync_checkpoint from stamp.utils.errors import StampPipelineError, StampValidationError from stamp.utils.io import archive_and_remove_directory, load_cached_picks, load_tomogram_manifests, write_sidecar from stamp.utils.log import log @@ -219,6 +220,20 @@ def run_pick( if normalise_per_vesicle and not vesicle_labels_mrc_by_tomogram: raise StampPipelineError('--normalise-per-vesicle requires --vesicle-labels-mrc to resolve to at least one tomogram') + # everything that changes raw picks; consensus, seed, beam filter, workers and plots are left out so they can change between runs + fingerprint = { + 'pickers': list(picker_names), + 'segmentation': signatures({m.tomogram_id: m.segmentation_path for m in manifests}), + 'raw': signatures({m.tomogram_id: m.raw_tomogram_path for m in manifests}), + 'vesicle_labels': signatures(vesicle_labels_mrc_by_tomogram), + 'normalise_per_vesicle': normalise_per_vesicle, + 'voxel_size_angstrom': resolved_voxel_size_angstrom, + 'picker_params': extra_params, + 'backend': backend, + } + if sync_checkpoint(output_dir / 'raw', fingerprint): + log.info('Found matching partial output, resuming') + picks_by_picker: dict[str, list[RawPick]] = {} for picker_name in picker_names: adapter = _select_adapter(picker_name, backend) diff --git a/src/stamp/commands/refine.py b/src/stamp/commands/refine.py index f5ff93c..6bdaaa9 100644 --- a/src/stamp/commands/refine.py +++ b/src/stamp/commands/refine.py @@ -20,6 +20,7 @@ ) from stamp.run.state import stage_dir from stamp.schemas.particles import ClassAssignment, ParticleSet +from stamp.utils.checkpoint import file_signature, signatures, sync_checkpoint from stamp.utils.errors import StampAdapterError, StampPipelineError, StampValidationError from stamp.utils.io import resolve_directory_voxel_size_angstrom, write_sidecar from stamp.utils.log import log @@ -36,6 +37,14 @@ def _seed_reference(class_averages_dir: Path, class_id: str, half: str) -> Path: raise StampPipelineError(f'No class average for {class_id} half {half} in {class_averages_dir}') return matches[0] +# _half_map: a half-map finished by an earlier run, still in its half directory or already moved to the class directory, else None +def _half_map(tree: dict[str, Path], half: str) -> Path | None: + for directory in (tree[half], tree['combined']): + candidate = directory / f'final_{half}.mrc' + if candidate.is_file(): + return candidate + return None + # _with_inplane: return a copy of the particle with the estimated in-plane roll folded into its orientation def _with_inplane(particle, angle_by_particle: dict[str, float]): angle = angle_by_particle.get(particle.particle_id) @@ -115,13 +124,34 @@ def run_refine( half_a, half_b = split_class_by_half(target, assignments, particle_set.particles) half_a = [_with_inplane(p, angle_by_particle) for p in half_a] half_b = [_with_inplane(p, angle_by_particle) for p in half_b] - tree = refine_output_tree(output_dir, target) reference_a = _seed_reference(class_averages_dir, target, 'A') reference_b = _seed_reference(class_averages_dir, target, 'B') assert_distinct_references(reference_a, reference_b) - final_a = _run_half(adapter, runner, half_a, reference_a, tree['A'], parameters, backend) - final_b = _run_half(adapter, runner, half_b, reference_b, tree['B'], parameters, backend) + # everything that changes a half-map; mask, threshold and plots only affect the FSC that always reruns, so they can change between runs + fingerprint = { + 'particles': file_signature(particles, content=True), + 'class_assignments': file_signature(class_assignments, content=True), + 'references': signatures({'A': reference_a, 'B': reference_b}), + 'tool': tool, + 'iterations': iterations, + 'voxel_size_angstrom': voxel_size_angstrom, + 'backend': backend, + } + if sync_checkpoint(output_dir / target, fingerprint): # the class directory is both output and work dir + log.info(f'{target}: found matching partial output, resuming') + tree = refine_output_tree(output_dir, target) + + final_a = _half_map(tree, 'A') + if final_a is None: + final_a = _run_half(adapter, runner, half_a, reference_a, tree['A'], parameters, backend) + else: + log.info(f'{target}: half A loaded from checkpoint') + final_b = _half_map(tree, 'B') + if final_b is None: + final_b = _run_half(adapter, runner, half_b, reference_b, tree['B'], parameters, backend) + else: + log.info(f'{target}: half B loaded from checkpoint') with mrcfile.open(str(final_a), permissive=True) as mrc: map_a = np.asarray(mrc.data, dtype=np.float32) diff --git a/src/stamp/decoy/generate.py b/src/stamp/decoy/generate.py index 72aa2a9..71fcc19 100644 --- a/src/stamp/decoy/generate.py +++ b/src/stamp/decoy/generate.py @@ -23,6 +23,7 @@ from stamp.schemas.manifest import TomogramManifest from stamp.schemas.particles import Particle, ParticleSet from stamp.schemas.picks import RawPick +from stamp.utils.checkpoint import atomic_save_npz from stamp.utils.errors import StampValidationError from stamp.utils.io import cache_tomogram_picks, load_cached_picks from stamp.utils.log import log @@ -51,6 +52,23 @@ def _adaptive_decoy_count(n_real_picks: int, *, target_ratio: float = 1.0) -> in def _score_one_tomogram(job: _RejectedSurfaceJob) -> _ScoreSurfaceResult: return _score_surface(job.manifest, job.config) +# _SCORE_FIELDS: names under which _score_surface's six arrays are stored +_SCORE_FIELDS = ('points', 'normals', 'scores', 'winning_window', 'mean_scores', 'profile_scores') + +# _save_cached_scores: checkpoint one tomogram's surface scores +def _save_cached_scores(scores_dir: Path, tomogram_id: str, result: _ScoreSurfaceResult) -> None: + atomic_save_npz(scores_dir / f'{tomogram_id}.npz', **dict(zip(_SCORE_FIELDS, result))) + +# _load_cached_scores: one tomogram's checkpointed surface scores, or None when absent or unreadable +def _load_cached_scores(scores_dir: Path | None, tomogram_id: str) -> _ScoreSurfaceResult | None: + if scores_dir is None: + return None + try: + with np.load(scores_dir / f'{tomogram_id}.npz') as cached: + return tuple(cached[name] for name in _SCORE_FIELDS) # type: ignore[return-value] + except (OSError, ValueError, KeyError): + return None + # generate_rejected_surface_decoys: sample decoys from surface points the native picker scored and rejected def generate_rejected_surface_decoys( real_particle_set: ParticleSet, @@ -74,11 +92,15 @@ def generate_rejected_surface_decoys( cache_dir = output_dir / 'raw' / METHOD_REJECTED_SURFACE if output_dir is not None else None tomogram_ids = [] + scores_dir = cache_dir / 'scores' if cache_dir is not None else None + # run surface scoring in a pool but keep rng-driven sampling sequential and in manifest order so results stay reproducible for a given seed result_by_tomogram: dict[str, _ScoreSurfaceResult] = {} def _on_success(job: _RejectedSurfaceJob, result: _ScoreSurfaceResult) -> None: result_by_tomogram[job.manifest.tomogram_id] = result + if scores_dir is not None: + _save_cached_scores(scores_dir, job.manifest.tomogram_id, result) def _on_error(job: _RejectedSurfaceJob, exc: Exception) -> None: if isinstance(exc, StampValidationError): @@ -86,7 +108,15 @@ def _on_error(job: _RejectedSurfaceJob, exc: Exception) -> None: else: raise exc - jobs = [_RejectedSurfaceJob(manifest, config) for manifest in manifests] + jobs: list[_RejectedSurfaceJob] = [] + for manifest in manifests: + cached = _load_cached_scores(scores_dir, manifest.tomogram_id) + if cached is None: + jobs.append(_RejectedSurfaceJob(manifest, config)) + else: + result_by_tomogram[manifest.tomogram_id] = cached + if len(jobs) < len(manifests): + log.info(f'Resuming rejected-surface decoys: {len(manifests) - len(jobs)} tomogram(s) loaded from checkpoint') run_parallel(jobs, _score_one_tomogram, max_workers=n_workers, label='decoy-surface-score', on_success=_on_success, on_error=_on_error) for manifest in manifests: @@ -161,6 +191,21 @@ def _extract_surface_for_manifest(manifest: TomogramManifest) -> tuple[tuple[int vertices, normals = extract_surface(segmentation) return segmentation.shape, vertices, normals +# _save_cached_surface: checkpoint one tomogram's extracted surface +def _save_cached_surface(surfaces_dir: Path, tomogram_id: str, surface: tuple[tuple[int, ...], np.ndarray, np.ndarray]) -> None: + shape, vertices, normals = surface + atomic_save_npz(surfaces_dir / f'{tomogram_id}.npz', shape=np.array(shape), vertices=vertices, normals=normals) + +# _load_cached_surface: one tomogram's checkpointed surface, or None when absent or unreadable +def _load_cached_surface(surfaces_dir: Path | None, tomogram_id: str) -> tuple[tuple[int, ...], np.ndarray, np.ndarray] | None: + if surfaces_dir is None: + return None + try: + with np.load(surfaces_dir / f'{tomogram_id}.npz') as cached: + return tuple(int(size) for size in cached['shape']), cached['vertices'], cached['normals'] + except (OSError, ValueError, KeyError): + return None + # generate_shifted_decoys: displace each real pick by a large random vector away from surface and picks def generate_shifted_decoys( real_particle_set: ParticleSet, @@ -178,21 +223,38 @@ def generate_shifted_decoys( ) -> ParticleSet | None: log.progress('Generating shifted decoys') rng = np.random.default_rng(seed) + surfaces_dir = output_dir / 'raw' / METHOD_SHIFTED / 'surfaces' if output_dir is not None else None shape_by_tomogram = {} surface_by_tomogram: dict[str, tuple[cKDTree, np.ndarray]] = {} - def _on_success(manifest: TomogramManifest, result: tuple[tuple[int, ...], np.ndarray, np.ndarray] | None) -> None: - if result is None: - return + + # _register: index one tomogram's surface for the sampling loop below + def _register(manifest: TomogramManifest, result: tuple[tuple[int, ...], np.ndarray, np.ndarray]) -> None: shape, vertices, normals = result shape_by_tomogram[manifest.tomogram_id] = shape surface_by_tomogram[manifest.tomogram_id] = (cKDTree(vertices[:, ::-1]), normals[:, ::-1]) + + def _on_success(manifest: TomogramManifest, result: tuple[tuple[int, ...], np.ndarray, np.ndarray] | None) -> None: + if result is None: + return + _register(manifest, result) + if surfaces_dir is not None: + _save_cached_surface(surfaces_dir, manifest.tomogram_id, result) def _on_error(manifest: TomogramManifest, exc: Exception) -> None: if isinstance(exc, ValueError): log.warning(f'{manifest.tomogram_id}: skipped ({exc})') else: raise exc - run_parallel(manifests, _extract_surface_for_manifest, max_workers=n_workers, label='decoy-surface-extract', on_success=_on_success, on_error=_on_error) + pending: list[TomogramManifest] = [] + for manifest in manifests: + cached = _load_cached_surface(surfaces_dir, manifest.tomogram_id) + if cached is None: + pending.append(manifest) + else: + _register(manifest, cached) + if len(pending) < len(manifests): + log.info(f'Resuming shifted decoys: {len(manifests) - len(pending)} tomogram(s) loaded from checkpoint') + run_parallel(pending, _extract_surface_for_manifest, max_workers=n_workers, label='decoy-surface-extract', on_success=_on_success, on_error=_on_error) real_by_tomogram: dict[str, list[tuple[float, float, float]]] = {} for particle in real_particle_set.particles: @@ -271,20 +333,11 @@ class _SyntheticNoiseJob: n_decoys_per_tomogram: int cache_dir: Path | None = None -# _generate_one_synthetic_noise_tomogram: multiprocessing worker entry point +# _generate_one_synthetic_noise_tomogram: multiprocessing worker entry point, reusing a finished volume from the checkpoint def _generate_one_synthetic_noise_tomogram(job: _SyntheticNoiseJob) -> tuple[TomogramManifest, list[RawPick]]: - rng = np.random.default_rng(job.seed + job.index) tomogram_id = f'decoy-noise-{job.index:03d}' - volume = rng.normal(0.0, 1.0, size=job.tomogram_shape).astype(np.float32) - volume[job.shell > 0] += job.config.density_sign * 2.0 - segmentation_path = job.segmentation_dir / f'{tomogram_id}.mrc' tomogram_path = job.tomogram_dir / f'{tomogram_id}.mrc' - with mrcfile.new(segmentation_path, overwrite=True) as mrc: - mrc.set_data(job.shell) - with mrcfile.new(tomogram_path, overwrite=True) as mrc: - mrc.set_data(volume) - manifest = TomogramManifest( tomogram_id=tomogram_id, segmentation_path=segmentation_path, @@ -293,6 +346,20 @@ def _generate_one_synthetic_noise_tomogram(job: _SyntheticNoiseJob) -> tuple[Tom is_decoy=True, ) + # the pick cache is written last and atomically, so its presence means the volumes and picks are complete + if job.cache_dir is not None and (job.cache_dir / f'{tomogram_id}.json').is_file() and segmentation_path.is_file() and tomogram_path.is_file(): + log.debug(f'{tomogram_id}: loaded from checkpoint') + return manifest, load_cached_picks(job.cache_dir, [tomogram_id]) + + rng = np.random.default_rng(job.seed + job.index) + volume = rng.normal(0.0, 1.0, size=job.tomogram_shape).astype(np.float32) + volume[job.shell > 0] += job.config.density_sign * 2.0 + + with mrcfile.new(segmentation_path, overwrite=True) as mrc: + mrc.set_data(job.shell) + with mrcfile.new(tomogram_path, overwrite=True) as mrc: + mrc.set_data(volume) + shell_voxels = np.argwhere(job.shell > 0) chosen = rng.choice( shell_voxels.shape[0], diff --git a/src/stamp/utils/checkpoint.py b/src/stamp/utils/checkpoint.py index e4e1c45..1b93b90 100644 --- a/src/stamp/utils/checkpoint.py +++ b/src/stamp/utils/checkpoint.py @@ -19,6 +19,16 @@ def file_signature(path: Path, *, content: bool = False) -> str: stat = path.stat() return f'{stat.st_size}:{stat.st_mtime_ns}' +# signatures: file_signature per role, sorted so the fingerprint is stable +def signatures(paths: dict[str, Path], *, content: bool = False) -> dict[str, str]: + return {role: file_signature(Path(path), content=content) for role, path in sorted(paths.items())} + +# directory_signatures: signatures of every file in directory matching pattern, keyed by filename; empty when directory is None +def directory_signatures(directory: Path | None, pattern: str = '*.mrc') -> dict[str, str]: + if directory is None: + return {} + return signatures({path.name: path for path in directory.glob(pattern)}) + # atomic_save_npz: write arrays via a temporary file so a partial file never looks complete def atomic_save_npz(path: Path, **arrays: np.ndarray) -> None: path.parent.mkdir(parents=True, exist_ok=True) @@ -36,6 +46,7 @@ def atomic_write_json(path: Path, payload: object) -> None: # sync_checkpoint: keep work_dir when its stored fingerprint matches exactly, else wipe it; returns True when resuming def sync_checkpoint(work_dir: Path, fingerprint: dict) -> bool: + fingerprint = json.loads(json.dumps(fingerprint, default=str)) # compare as it reads back: tuples become lists, paths strings path = work_dir / _CHECKPOINT_NAME try: stored = json.loads(path.read_text()) diff --git a/src/stamp/utils/io.py b/src/stamp/utils/io.py index 709e715..efe615f 100644 --- a/src/stamp/utils/io.py +++ b/src/stamp/utils/io.py @@ -324,17 +324,23 @@ def print_toml(path: Path) -> None: console.print(build_toml_tree(data, path.name)) console.print() -# cache_tomogram_picks: write one tomogram's picks to /.json +# cache_tomogram_picks: atomically write one tomogram's picks to /.json def cache_tomogram_picks(cache_dir: Path, tomogram_id: str, picks: list[RawPick]) -> None: - cache_dir.mkdir(parents=True, exist_ok=True) - (cache_dir / f'{tomogram_id}.json').write_text(json.dumps([pick.model_dump() for pick in picks], indent=2)) + from stamp.utils.checkpoint import atomic_write_json # lazy: log imports io and checkpoint imports log + atomic_write_json(cache_dir / f'{tomogram_id}.json', [pick.model_dump() for pick in picks]) -# load_cached_picks: read back per-tomogram JSON pick caches written by cache_tomogram_picks +# load_cached_picks: read back per-tomogram JSON pick caches written by cache_tomogram_picks, skipping unreadable ones def load_cached_picks(cache_dir: Path, tomogram_ids: list[str]) -> list[RawPick]: + from stamp.utils.log import log picks: list[RawPick] = [] for tomogram_id in tomogram_ids: cache_path = cache_dir / f'{tomogram_id}.json' if not cache_path.exists(): continue - picks.extend(RawPick(**raw) for raw in json.loads(cache_path.read_text())) + try: + raw_picks = json.loads(cache_path.read_text()) + except json.JSONDecodeError: + log.warning(f'{cache_path} is not valid JSON, ignoring it') + continue + picks.extend(RawPick(**raw) for raw in raw_picks) return picks diff --git a/tests/integration/test_decoy_command.py b/tests/integration/test_decoy_command.py index 8ffb183..ccb4bf1 100644 --- a/tests/integration/test_decoy_command.py +++ b/tests/integration/test_decoy_command.py @@ -33,10 +33,11 @@ def _write_pair(seg_dir: Path, raw_dir: Path, tomogram_id: str) -> None: with mrcfile.new(raw_dir / f'{tomogram_id}.mrc', overwrite=True) as mrc: mrc.set_data(tomogram) -# _run_pick: run `stamp pick` on a fresh vesicle pair and return its paths -def _run_pick(tmp_path: Path) -> tuple[Path, Path, Path]: +# _run_pick: run `stamp pick` on fresh vesicle pairs and return its paths +def _run_pick(tmp_path: Path, n_tomograms: int = 1) -> tuple[Path, Path, Path]: seg_dir, raw_dir, pick_out = tmp_path / 'seg', tmp_path / 'raw', tmp_path / 'stamp' / 'pick' - _write_pair(seg_dir, raw_dir, 'tomo000') + for index in range(n_tomograms): + _write_pair(seg_dir, raw_dir, f'tomo{index:03d}') result = runner.invoke( stamp_app, [ @@ -54,6 +55,44 @@ def _run_pick(tmp_path: Path) -> tuple[Path, Path, Path]: assert result.exit_code == 0, result.output return seg_dir, raw_dir, pick_out / 'particle_set.json' +# _positions: decoy positions from a decoy_particle_set.json, in file order +def _positions(path: Path) -> list[tuple[float, float, float]]: + return [p.position for p in ParticleSet.model_validate(json.loads(path.read_text())).particles] + +# _rejected_surface_args: CLI arguments for a rejected-surface decoy run writing under out_dir +def _rejected_surface_args(tmp_path: Path, out_dir: Path, seg_dir: Path, raw_dir: Path, particle_set_path: Path) -> list[str]: + return [ + '-d', str(tmp_path), + 'decoy', + '--method', 'rejected-surface', + '--real-particle-set', str(particle_set_path), + '--seg-dir', str(seg_dir), + '--raw-dir', str(raw_dir), + '--out-dir', str(out_dir), + '--voxel-size-a', '10.0', + '--picker-params', json.dumps({'n_mad': 2.5, 'offset_windows_angstrom': [[20.0, 80.0]]}), + '--n-decoys-per-tomogram', '20', + '--seed', '1', + '--n-processes', '1', + ] + +# _shifted_args: CLI arguments for a shifted decoy run writing under out_dir +def _shifted_args(tmp_path: Path, out_dir: Path, seg_dir: Path, raw_dir: Path, particle_set_path: Path) -> list[str]: + return [ + '-d', str(tmp_path), + 'decoy', + '--method', 'shifted', + '--real-particle-set', str(particle_set_path), + '--seg-dir', str(seg_dir), + '--raw-dir', str(raw_dir), + '--out-dir', str(out_dir), + '--voxel-size-a', '10.0', + '--min-shift-a', '100.0', + '--max-shift-a', '180.0', + '--seed', '2', + '--n-processes', '1', + ] + # TestDecoyCommand: class containing unit tests for test_decoy_command.py class TestDecoyCommand: def test_decoy_rejected_surface_end_to_end(self, tmp_path: Path) -> None: @@ -194,3 +233,88 @@ def test_decoy_rejects_undersized_set(self, tmp_path: Path) -> None: assert isinstance(result.exception, StampPipelineError) assert 'less than half the size of the real set' in str(result.exception) assert not (tmp_path / 'stamp' / 'decoy' / 'decoy_particle_set.json').exists() + + def test_decoy_synthetic_noise_reuses_finished_volumes(self, tmp_path: Path) -> None: + '''A rerun over a kept raw dir reuses finished volumes and gives identical decoys.''' + args = [ + '-d', str(tmp_path), + 'decoy', + '--method', 'synthetic-noise', + '--out-dir', str(tmp_path), + '--voxel-size-a', '10.0', + '--n-synthetic-tomograms', '2', + '--synthetic-shape', '40,40,40', + '--n-decoys-per-tomogram', '5', + '--seed', '3', + '--n-processes', '1', + '--keep-raw', + ] + output_dir = tmp_path / 'stamp' / 'decoy' + assert runner.invoke(stamp_app, args).exit_code == 0 + volume = output_dir / 'tomograms' / 'decoy-noise-000.mrc' + modified = volume.stat().st_mtime_ns + first = _positions(output_dir / 'decoy_particle_set.json') + + assert runner.invoke(stamp_app, args).exit_code == 0 + assert volume.stat().st_mtime_ns == modified + assert _positions(output_dir / 'decoy_particle_set.json') == first + + def test_decoy_rejected_surface_resumes_after_interruption(self, tmp_path: Path, monkeypatch) -> None: + '''Scoring resumes from finished tomograms and the decoys match an uninterrupted run.''' + from stamp.decoy import generate + seg_dir, raw_dir, particle_set_path = _run_pick(tmp_path, n_tomograms=3) + real = generate._score_surface + scored: list[str] = [] + state = {'fail': True} + + # flaky: fail on the second tomogram until told otherwise, else score normally + def flaky(manifest, config): + if state['fail'] and len(scored) == 1: + raise RuntimeError('boom') + scored.append(manifest.tomogram_id) + return real(manifest, config) + + monkeypatch.setattr(generate, '_score_surface', flaky) + args = _rejected_surface_args(tmp_path, tmp_path, seg_dir, raw_dir, particle_set_path) + assert runner.invoke(stamp_app, args).exit_code != 0 + finished = list(scored) + assert len(finished) == 1 + + state['fail'] = False + scored.clear() + assert runner.invoke(stamp_app, args).exit_code == 0 + assert len(scored) == 2 and not set(scored) & set(finished) + + clean = _rejected_surface_args(tmp_path, tmp_path / 'clean', seg_dir, raw_dir, particle_set_path) + assert runner.invoke(stamp_app, clean).exit_code == 0 + assert _positions(tmp_path / 'stamp' / 'decoy' / 'decoy_particle_set.json') == _positions(tmp_path / 'clean' / 'stamp' / 'decoy' / 'decoy_particle_set.json') + + def test_decoy_shifted_resumes_after_interruption(self, tmp_path: Path, monkeypatch) -> None: + '''Surface extraction resumes from finished tomograms and the decoys match an uninterrupted run.''' + from stamp.decoy import generate + seg_dir, raw_dir, particle_set_path = _run_pick(tmp_path, n_tomograms=3) + real = generate._extract_surface_for_manifest + extracted: list[str] = [] + state = {'fail': True} + + # flaky: fail on the second tomogram until told otherwise, else extract normally + def flaky(manifest): + if state['fail'] and len(extracted) == 1: + raise RuntimeError('boom') + extracted.append(manifest.tomogram_id) + return real(manifest) + + monkeypatch.setattr(generate, '_extract_surface_for_manifest', flaky) + args = _shifted_args(tmp_path, tmp_path, seg_dir, raw_dir, particle_set_path) + assert runner.invoke(stamp_app, args).exit_code != 0 + finished = list(extracted) + assert len(finished) == 1 + + state['fail'] = False + extracted.clear() + assert runner.invoke(stamp_app, args).exit_code == 0 + assert len(extracted) == 2 and not set(extracted) & set(finished) + + clean = _shifted_args(tmp_path, tmp_path / 'clean', seg_dir, raw_dir, particle_set_path) + assert runner.invoke(stamp_app, clean).exit_code == 0 + assert _positions(tmp_path / 'stamp' / 'decoy' / 'decoy_particle_set.json') == _positions(tmp_path / 'clean' / 'stamp' / 'decoy' / 'decoy_particle_set.json') diff --git a/tests/integration/test_identify_command.py b/tests/integration/test_identify_command.py index 5b57d4e..25eeee6 100644 --- a/tests/integration/test_identify_command.py +++ b/tests/integration/test_identify_command.py @@ -67,3 +67,50 @@ def test_identify_end_to_end(self, tmp_path): assert any(line.startswith('DECOY CONTROL:') for line in report.splitlines()) assert (out / 'decoy_control.json').is_file() assert (out / 'params.toml').is_file() + + def test_identify_resumes_after_interruption(self, tmp_path, monkeypatch): + from stamp.commands import identify + classes = tmp_path / 'classes' + classes.mkdir() + _write_average(classes / 'c00_halfA_n40.mrc', 21, 2.0) + _write_average(classes / 'c01_halfA_n30.mrc', 21, 5.0) + _write_pdb(tmp_path / 'small.pdb', 6) + _write_pdb(tmp_path / 'large.pdb', 60) + (tmp_path / 'candidates.yaml').write_text( + 'candidates:\n' + ' - name: small\n structure_path: small.pdb\n' + ' - name: large\n structure_path: large.pdb\n' + ) + real = identify._fit_one + fitted: list[tuple[str, str]] = [] + state = {'fail': True} + + # flaky: fail on the third fit until told otherwise, else fit normally + def flaky(job): + if state['fail'] and len(fitted) == 2: + raise RuntimeError('boom') + fitted.append((job.class_id, job.candidate.name)) + return real(job) + + monkeypatch.setattr(identify, '_fit_one', flaky) + args = [ + '-d', str(tmp_path), + 'identify', + '--classes', str(classes), + '--candidates', str(tmp_path / 'candidates.yaml'), + '--output-dir', str(tmp_path), + '--resolution', '30.0', + '-n', '1', + ] + assert runner.invoke(stamp_app, args).exit_code != 0 + finished = list(fitted) + assert len(finished) == 2 + + state['fail'] = False + fitted.clear() + result = runner.invoke(stamp_app, args) + assert result.exit_code == 0, result.output + assert len(fitted) == 2 and not set(fitted) & set(finished) + out = tmp_path / 'stamp' / 'identify' + assert {row['cluster_id'] for row in json.loads((out / 'identification.json').read_text())} == {'c00', 'c01'} + assert not (out / 'raw').exists() diff --git a/tests/integration/test_pick_command.py b/tests/integration/test_pick_command.py index 3755a94..f7ec01e 100644 --- a/tests/integration/test_pick_command.py +++ b/tests/integration/test_pick_command.py @@ -36,6 +36,21 @@ def _write_pair(seg_dir: Path, raw_dir: Path, tomogram_id: str, *, voxel_size: f if voxel_size is not None: mrc.voxel_size = voxel_size +# _pick_args: CLI arguments for a native pick run over the given directories +def _pick_args(tmp_path: Path, seg_dir: Path, raw_dir: Path, n_mad: float) -> list[str]: + return [ + '-d', str(tmp_path), + 'pick', + '--seg-dir', str(seg_dir), + '--raw-dir', str(raw_dir), + '--out-dir', str(tmp_path), + '--voxel-size-a', '10.0', + '--picker-params', json.dumps({'stamp-native': {'n_mad': n_mad, 'offset_windows_angstrom': [[20.0, 80.0]]}}), + '--backend', 'local', + '--n-processes', '1', + 'stamp-native', + ] + # TestPickCommand: class containing tests for pick command class TestPickCommand: # check `stamp pick` runs the native picker and writes outputs @@ -167,4 +182,55 @@ def test_pick_warns_and_uses_most_common_voxel_size(self, tmp_path: Path) -> Non ], ) assert result.exit_code == 0, result.output - assert 'Multiple voxel sizes were found' in result.output \ No newline at end of file + assert 'Multiple voxel sizes were found' in result.output + # check an interrupted pick resumes, picking only the tomograms that had not finished + def test_pick_resumes_after_interruption(self, tmp_path: Path, monkeypatch) -> None: + from stamp.adapters import native + seg_dir, raw_dir = tmp_path / 'seg', tmp_path / 'raw' + for tomogram_id in ('tomo000', 'tomo001', 'tomo002'): + _write_pair(seg_dir, raw_dir, tomogram_id) + real = native.pick_tomogram + picked: list[str] = [] + state = {'fail': True} + + # flaky: fail on tomo001 until told otherwise, else pick normally + def flaky(segmentation_path, tomogram_path, tomogram_id, config): + if state['fail'] and tomogram_id == 'tomo001': + raise RuntimeError('boom') + picked.append(tomogram_id) + return real(segmentation_path, tomogram_path, tomogram_id, config) + + monkeypatch.setattr(native, 'pick_tomogram', flaky) + args = _pick_args(tmp_path, seg_dir, raw_dir, 2.5) + assert runner.invoke(stamp_app, args).exit_code != 0 + assert picked == ['tomo000'] # manifests are processed in sorted order + + state['fail'] = False + picked.clear() + result = runner.invoke(stamp_app, args) + assert result.exit_code == 0, result.output + assert picked == ['tomo001', 'tomo002'] + + # check changed picker parameters discard the checkpoint + def test_pick_restarts_when_picker_params_change(self, tmp_path: Path, monkeypatch) -> None: + from stamp.adapters import native + seg_dir, raw_dir = tmp_path / 'seg', tmp_path / 'raw' + for tomogram_id in ('tomo000', 'tomo001'): + _write_pair(seg_dir, raw_dir, tomogram_id) + real = native.pick_tomogram + picked: list[str] = [] + state = {'fail': True} + + # flaky: fail on tomo001 until told otherwise, else pick normally + def flaky(segmentation_path, tomogram_path, tomogram_id, config): + if state['fail'] and tomogram_id == 'tomo001': + raise RuntimeError('boom') + picked.append(tomogram_id) + return real(segmentation_path, tomogram_path, tomogram_id, config) + + monkeypatch.setattr(native, 'pick_tomogram', flaky) + runner.invoke(stamp_app, _pick_args(tmp_path, seg_dir, raw_dir, 2.5)) + state['fail'] = False + picked.clear() + runner.invoke(stamp_app, _pick_args(tmp_path, seg_dir, raw_dir, 2.0)) # exit code irrelevant, only what ran matters + assert picked == ['tomo000', 'tomo001'] diff --git a/tests/integration/test_refine_command.py b/tests/integration/test_refine_command.py index 847466b..1c0a9f9 100644 --- a/tests/integration/test_refine_command.py +++ b/tests/integration/test_refine_command.py @@ -60,7 +60,7 @@ def _fixture(tmp_path, mixed=False): (tmp_path / 'tomo').mkdir() return stage_d -def _invoke(tmp_path, stage_d): +def _invoke(tmp_path, stage_d, *extra): return runner.invoke(stamp_app, [ '-d', str(tmp_path), 'refine', @@ -72,6 +72,7 @@ def _invoke(tmp_path, stage_d): '--out-dir', tmp_path, '--voxel-size-a', '3.0', '--backend', 'mock', + *extra, ]) # TestRefineCommand: class containing tests for `stamp refine` @@ -93,3 +94,50 @@ def test_refine_rejects_half_set_mixed_class(self, tmp_path): assert result.exit_code != 0 out = tmp_path / 'stamp' / 'refine' assert not (out / 'c00' / 'halfA' / 'particles.star').exists() + + def test_refine_resumes_unfinished_half(self, tmp_path, monkeypatch): + from stamp.commands import refine + stage_d = _fixture(tmp_path) + real = refine._run_half + ran: list[str] = [] + state = {'fail': True} + + # flaky: fail on half B until told otherwise, else refine normally + def flaky(adapter, runner, particles, reference, workdir, parameters, backend): + if state['fail'] and workdir.name == 'halfB': + raise RuntimeError('boom') + ran.append(workdir.name) + return real(adapter, runner, particles, reference, workdir, parameters, backend) + + monkeypatch.setattr(refine, '_run_half', flaky) + assert _invoke(tmp_path, stage_d).exit_code != 0 + assert ran == ['halfA'] + + state['fail'] = False + ran.clear() + result = _invoke(tmp_path, stage_d) + assert result.exit_code == 0, result.output + assert ran == ['halfB'] + assert (tmp_path / 'stamp' / 'refine' / 'c00' / 'final_A.mrc').is_file() + + def test_refine_reuses_finished_halves_until_parameters_change(self, tmp_path, monkeypatch): + from stamp.commands import refine + stage_d = _fixture(tmp_path) + real = refine._run_half + ran: list[str] = [] + + # counting: record which half ran, then refine normally + def counting(adapter, runner, particles, reference, workdir, parameters, backend): + ran.append(workdir.name) + return real(adapter, runner, particles, reference, workdir, parameters, backend) + + monkeypatch.setattr(refine, '_run_half', counting) + assert _invoke(tmp_path, stage_d, '--iterations', '2').exit_code == 0 + assert ran == ['halfA', 'halfB'] + + ran.clear() + assert _invoke(tmp_path, stage_d, '--iterations', '2').exit_code == 0 + assert ran == [] # both halves reused, FSC recomputed + + assert _invoke(tmp_path, stage_d, '--iterations', '3').exit_code == 0 + assert ran == ['halfA', 'halfB'] # changed iterations wipe the class directory diff --git a/tests/unit/test_classify.py b/tests/unit/test_classify.py index c333aae..0d6e60c 100644 --- a/tests/unit/test_classify.py +++ b/tests/unit/test_classify.py @@ -14,6 +14,7 @@ label_to_cluster_id, match_clusters_across_halves, reduce_and_cluster, + reduce_features, reduce_and_cluster_shared, ) from stamp.classify.extract import ( @@ -357,17 +358,14 @@ def test_kmeans_respects_requested_cluster_count(self) -> None: def test_pca_components_capped_by_sample_count(self) -> None: '''N_components is capped at what the data supports.''' features = np.random.default_rng(0).normal(size=(5, 100)) - result = reduce_and_cluster( - features, ClusteringConfig(method='kmeans', n_components=50, n_clusters=2) - ) - assert result.embedding.shape[1] <= 4 + embedding, explained = reduce_features(features, ClusteringConfig(n_components=50)) + assert embedding.shape[1] <= 4 + assert explained.shape == (embedding.shape[1],) def test_too_few_particles_raises(self) -> None: '''A single particle raises a clear error.''' with pytest.raises(ValueError, match='Too few particles'): - reduce_and_cluster( - np.zeros((1, 10)), ClusteringConfig(method='kmeans', n_clusters=1) - ) + reduce_features(np.zeros((1, 10)), ClusteringConfig()) def test_hdbscan_below_min_cluster_size_raises_clearly(self) -> None: '''Too few particles for min_cluster_size raises.''' diff --git a/tests/unit/test_utils.py b/tests/unit/test_utils.py index 1f8e65b..9106966 100644 --- a/tests/unit/test_utils.py +++ b/tests/unit/test_utils.py @@ -348,3 +348,31 @@ def test_matching_fingerprint_resumes_and_mismatch_wipes(self, tmp_path: Path) - assert (raw / 'kept.npy').exists() assert sync_checkpoint(raw, {'box': 13}) is False assert not (raw / 'kept.npy').exists() + + def test_tuple_and_path_values_still_resume(self, tmp_path: Path) -> None: + from stamp.utils.checkpoint import sync_checkpoint + raw = tmp_path / 'raw' + fingerprint = {'window': (20.0, 80.0), 'path': tmp_path / 'x'} + assert sync_checkpoint(raw, fingerprint) is False + assert sync_checkpoint(raw, fingerprint) is True + + def test_signatures_change_with_file_content(self, tmp_path: Path) -> None: + from stamp.utils.checkpoint import directory_signatures, signatures + (tmp_path / 'a.mrc').write_text('x') + first = directory_signatures(tmp_path) + assert set(first) == {'a.mrc'} + assert directory_signatures(None) == {} + (tmp_path / 'a.mrc').write_text('longer') + assert directory_signatures(tmp_path) != first + assert signatures({'a': tmp_path / 'a.mrc'}, content=True) == signatures({'a': tmp_path / 'a.mrc'}, content=True) + +# TestPickCache: class containing unit tests for cache_tomogram_picks and load_cached_picks in src/stamp/utils/io.py +class TestPickCache: + def test_round_trip_and_truncated_file_is_skipped(self, tmp_path: Path) -> None: + from stamp.schemas.picks import RawPick + from stamp.utils.io import cache_tomogram_picks, load_cached_picks + pick = RawPick(tomogram_id='t0', position=(1.0, 2.0, 3.0), source_picker='x') + cache_tomogram_picks(tmp_path / 'cache', 't0', [pick]) + (tmp_path / 'cache' / 't1.json').write_text('[{"tomogram_id": ') # truncated + loaded = load_cached_picks(tmp_path / 'cache', ['t0', 't1']) + assert [p.tomogram_id for p in loaded] == ['t0'] diff --git a/uv.lock b/uv.lock index 25b3090..38fa4a2 100644 --- a/uv.lock +++ b/uv.lock @@ -85,6 +85,20 @@ wheels = [ { url = "https://files.pythonhosted.org/packages/e7/05/c19819d5e3d95294a6f5947fb9b9629efb316b96de511b418c53d245aae6/cycler-0.12.1-py3-none-any.whl", hash = "sha256:85cef7cff222d8644161529808465972e51340599459b8ac3ccbac5a854e0d30", size = 8321, upload-time = "2023-10-07T05:32:16.783Z" }, ] +[[package]] +name = "fast-hdbscan" +version = "0.3.2" +source = { registry = "https://pypi.org/simple" } +dependencies = [ + { name = "numba" }, + { name = "numpy" }, + { name = "scikit-learn" }, +] +sdist = { url = "https://files.pythonhosted.org/packages/2b/3b/235c47dd8282610522f9fa6869ad8b896c2d5927f82039f66d547229fc07/fast_hdbscan-0.3.2.tar.gz", hash = "sha256:248e09202eda04da4b84dd819de3cc714469912205bb447ad4c7136db580f205", size = 59800, upload-time = "2026-04-07T16:43:33.12Z" } +wheels = [ + { url = "https://files.pythonhosted.org/packages/d0/92/34dac2dff0877d637535bd8852bc68323f155799521c9ee94c27c548426f/fast_hdbscan-0.3.2-py3-none-any.whl", hash = "sha256:619b42963935e6c3ab1e51de7895dd35260f3ef9ebe4627ff4317073fe51a667", size = 62642, upload-time = "2026-04-07T16:43:31.918Z" }, +] + [[package]] name = "fonttools" version = "4.64.0" @@ -246,6 +260,31 @@ wheels = [ { url = "https://files.pythonhosted.org/packages/8a/a1/8d812e53a5da1687abb10445275d41a8b13adb781bbf7196ddbcf8d88505/lazy_loader-0.5-py3-none-any.whl", hash = "sha256:ab0ea149e9c554d4ffeeb21105ac60bed7f3b4fd69b1d2360a4add51b170b005", size = 8044, upload-time = "2026-03-06T15:45:07.668Z" }, ] +[[package]] +name = "llvmlite" +version = "0.50.0" +source = { registry = "https://pypi.org/simple" } +sdist = { url = "https://files.pythonhosted.org/packages/11/c5/907cec40688a34eb489cded74d555e1ee4af8cf49d83e03dba2c2d4cfe27/llvmlite-0.50.0.tar.gz", hash = "sha256:f2a2cd6ec9ffcc1b7147dea0d7a49efebf17a2b434e0c2844fe175999d571eb4", size = 194522, upload-time = "2026-09-29T18:44:46.782Z" } +wheels = [ + { url = "https://files.pythonhosted.org/packages/b8/08/eecfccb51bc016de4c1fb69da815738076a186158fa61d3cae1458b8f44a/llvmlite-0.50.0-cp314-cp314-macosx_12_0_arm64.whl", hash = "sha256:ecdc9fae295da8ac793578a27020515e24d970513143efa227e696582aeb16e6", size = 40534277, upload-time = "2026-09-29T18:43:37.013Z" }, + { url = "https://files.pythonhosted.org/packages/9a/96/011ae57fb82e326a79da1c4767b8206502dbac041068b37f1fbe73893a55/llvmlite-0.50.0-cp314-cp314-manylinux_2_27_aarch64.manylinux_2_28_aarch64.whl", hash = "sha256:987600ce6f7bd6d808f4bb0ea61a8eff2fd17cf32355691e801eb0a65a7304f0", size = 58344485, upload-time = "2026-09-29T18:43:41.242Z" }, + { url = "https://files.pythonhosted.org/packages/5c/ed/54107648386edf3da7def03d42721c72279f6bc2e17b5274c18955dc5833/llvmlite-0.50.0-cp314-cp314-manylinux_2_27_x86_64.manylinux_2_28_x86_64.whl", hash = "sha256:33ddf12b1e12d7e551e1c1e6ca8087d0aacc931f480019eb33ef2ab77681da4d", size = 59696587, upload-time = "2026-09-29T18:43:46.132Z" }, + { url = "https://files.pythonhosted.org/packages/d1/af/b2e5f9ee84f05a794e62626d83a934e6fccc7a83740918a90cec85df2d6f/llvmlite-0.50.0-cp314-cp314-win_amd64.whl", hash = "sha256:7ae211012c6849528a5f7cd17a78d8b2421a2813c7b4184d6c0b2ffa89a7d296", size = 42986708, upload-time = "2026-09-29T18:43:51.123Z" }, + { url = "https://files.pythonhosted.org/packages/3b/df/6d9ac4237f78bc81e6778d87ec711c6e5ec0fac73f00907b149c414b48b5/llvmlite-0.50.0-cp314-cp314-win_arm64.whl", hash = "sha256:e94f9066f1257a9cef6c832e6c9de0f140e2bb150de2db39f657b2a5996e0f6b", size = 37441844, upload-time = "2026-09-29T18:43:55.097Z" }, + { url = "https://files.pythonhosted.org/packages/d6/23/0f9d73a3603fee0d32a0f66996e00964154f07681c0b0f9c7212e896cb2d/llvmlite-0.50.0-cp314-cp314t-macosx_12_0_arm64.whl", hash = "sha256:423c8d89d13f7eb4488933d5a86b0fa952927956298cfd0087f6753b5123b5df", size = 40534276, upload-time = "2026-09-29T18:43:59.379Z" }, + { url = "https://files.pythonhosted.org/packages/34/14/45f56e4cf192284ba6cb3020ed775d47dd9c69e7fb605f7523047ab16d7f/llvmlite-0.50.0-cp314-cp314t-manylinux_2_27_aarch64.manylinux_2_28_aarch64.whl", hash = "sha256:944133e9621d1dfbfdaf0fed3234b99f85e6ba27c38f4045acc8f8a5e699a5c0", size = 58344486, upload-time = "2026-09-29T18:44:03.923Z" }, + { url = "https://files.pythonhosted.org/packages/82/f8/45f08fe27bd96fa38a7199024d842d6ef502054f1f824b531d55cd533c81/llvmlite-0.50.0-cp314-cp314t-manylinux_2_27_x86_64.manylinux_2_28_x86_64.whl", hash = "sha256:a1d5b6eac064f201b4aa091030282e6f240d8d322dddd7381840731455c3e664", size = 59696589, upload-time = "2026-09-29T18:44:09.376Z" }, + { url = "https://files.pythonhosted.org/packages/90/68/e00620b48cd6fd71369877ddbfa000854450b843c3631be41226e8b8f7b1/llvmlite-0.50.0-cp314-cp314t-win_amd64.whl", hash = "sha256:d88c9b325f5fbefc79d95b1daa8fb96018c40bd2958103eea7334e6c8f17fb40", size = 42986716, upload-time = "2026-09-29T18:44:13.366Z" }, + { url = "https://files.pythonhosted.org/packages/4e/97/78e51381def071781a5ec9ead92e2a55562da5b78043566865e20f30be77/llvmlite-0.50.0-cp315-cp315-macosx_12_0_arm64.whl", hash = "sha256:3f490c0f4800c8ddeee6a607acd037497bf6508586804f4e2f11f53a1ee7fe2d", size = 40534277, upload-time = "2026-09-29T18:44:17.301Z" }, + { url = "https://files.pythonhosted.org/packages/61/83/1beb6169126cd1a8199bae88eb3a79e3be3dd609eb42896d8fa8c38b10c0/llvmlite-0.50.0-cp315-cp315-manylinux_2_27_aarch64.manylinux_2_28_aarch64.whl", hash = "sha256:d5447a6c39171368edfe28a71f605e6e3edd40a1dc31f5e5c9d50585718ae6d0", size = 58344486, upload-time = "2026-09-29T18:44:21.407Z" }, + { url = "https://files.pythonhosted.org/packages/7e/81/334b11c9ebc52ee5339fe401342b2dc856804996fec3abc5ad70ad053901/llvmlite-0.50.0-cp315-cp315-manylinux_2_27_x86_64.manylinux_2_28_x86_64.whl", hash = "sha256:f1ac2b9f699c46219fbbd66b304105f5e1b218f05ffac6fe03cd851f93718e58", size = 59696588, upload-time = "2026-09-29T18:44:25.755Z" }, + { url = "https://files.pythonhosted.org/packages/4f/c7/f06fe5d262f0cf0f0c85a85b0a4aaa07cbd85a56192861299fd659af4eb7/llvmlite-0.50.0-cp315-cp315-win_amd64.whl", hash = "sha256:51a4a716db98591f0a1bea34c6548cdb4017731ee5e678ded8cf842dca8af3c5", size = 42986709, upload-time = "2026-09-29T18:44:29.203Z" }, + { url = "https://files.pythonhosted.org/packages/be/f9/670bcb2a7214dcf35c48da581ac8d2949ff50255deb83e13c9cbbef46c05/llvmlite-0.50.0-cp315-cp315t-macosx_12_0_arm64.whl", hash = "sha256:e8cc203c1fd509131cd72b7554413d4a3e5527cc5558c5a7ebe19840018c57c1", size = 40534277, upload-time = "2026-09-29T18:44:32.967Z" }, + { url = "https://files.pythonhosted.org/packages/f3/21/3d108d6c9a87142927073fbc3d82d161f2dbfdeb046063a51edb196d1132/llvmlite-0.50.0-cp315-cp315t-manylinux_2_27_aarch64.manylinux_2_28_aarch64.whl", hash = "sha256:c7d4e2bbb29a860a6e85e22afdb96696241263942a5b214cac3e4b704e1d3abf", size = 58344488, upload-time = "2026-09-29T18:44:36.859Z" }, + { url = "https://files.pythonhosted.org/packages/6e/de/496d19b7a54acc487266ac7fa39d902cddf24998f5266b3aa499c8eacbd6/llvmlite-0.50.0-cp315-cp315t-manylinux_2_27_x86_64.manylinux_2_28_x86_64.whl", hash = "sha256:afd7b438c60e0f60c4368ec603bb9f20d938a203b5f59b80bbe50c749b4b2f16", size = 59696591, upload-time = "2026-09-29T18:44:40.642Z" }, + { url = "https://files.pythonhosted.org/packages/93/73/72553170eada174775d9a738c471c7be4ab3dc2c06368beeee89e002345c/llvmlite-0.50.0-cp315-cp315t-win_amd64.whl", hash = "sha256:4da0e8c6e6f144b433672a632f75d6b4da7bd4fdb5c3e9981d6ea6741319aeae", size = 42986722, upload-time = "2026-09-29T18:44:44.491Z" }, +] + [[package]] name = "loguru" version = "0.7.3" @@ -343,6 +382,31 @@ wheels = [ { url = "https://files.pythonhosted.org/packages/9e/c9/b2622292ea83fbb4ec318f5b9ab867d0a28ab43c5717bb85b0a5f6b3b0a4/networkx-3.6.1-py3-none-any.whl", hash = "sha256:d47fbf302e7d9cbbb9e2555a0d267983d2aa476bac30e90dfbe5669bd57f3762", size = 2068504, upload-time = "2025-12-08T17:02:38.159Z" }, ] +[[package]] +name = "numba" +version = "0.68.0" +source = { registry = "https://pypi.org/simple" } +dependencies = [ + { name = "llvmlite" }, + { name = "numpy" }, +] +sdist = { url = "https://files.pythonhosted.org/packages/4e/cd/e8280f9ffa30fea9fabc5341223701231fcc5d53a31f51419d42d4bec3a6/numba-0.68.0.tar.gz", hash = "sha256:8a781de54b980b98f43bff7f1093701b5f07c80d031c7cfa8a87493d8bf73f2d", size = 2855363, upload-time = "2026-09-30T15:05:44.721Z" } +wheels = [ + { url = "https://files.pythonhosted.org/packages/6e/71/a9031907dd0fba6cfce34004398a05f090b692be811dd1f38fdd874dd4e1/numba-0.68.0-cp314-cp314-macosx_12_0_arm64.whl", hash = "sha256:bfc890c9ca517823dfae0444595ef50d883ade9d3e17759d9a7650e5d128d950", size = 2760551, upload-time = "2026-09-30T15:05:15.753Z" }, + { url = "https://files.pythonhosted.org/packages/74/70/c03aebc576ded2204e5bde9b86b215f0590a81261af333d4239b9f0aed0f/numba-0.68.0-cp314-cp314-manylinux_2_27_aarch64.manylinux_2_28_aarch64.whl", hash = "sha256:34ccf54fd9c1d5f4ba00073b81bc492a681f5437c62917fe29813f457564e312", size = 3561561, upload-time = "2026-09-30T15:05:18.266Z" }, + { url = "https://files.pythonhosted.org/packages/3d/5f/2bd2fd4b99b0b5e76fea2f1fe149e05a7ec19a9a177758688bb82c7e3126/numba-0.68.0-cp314-cp314-manylinux_2_27_x86_64.manylinux_2_28_x86_64.whl", hash = "sha256:ea11c865265e39a6019e2f0fe62743825127b3b7bc4815916f5d5121fd9b262b", size = 3848766, upload-time = "2026-09-30T15:05:20.541Z" }, + { url = "https://files.pythonhosted.org/packages/0c/41/3e3528f3b0f9ffae69310d2e71f81ff74d272ee3b6c0600c4f4abaa31a80/numba-0.68.0-cp314-cp314-win_amd64.whl", hash = "sha256:9c03de7085f08ba11ab2444f252e822c14cee5fa02b73e84d5afd5e28b2bce0f", size = 2832584, upload-time = "2026-09-30T15:05:22.621Z" }, + { url = "https://files.pythonhosted.org/packages/8a/9d/1fe8be8f3a43d339222a4aed59be0b8f4920f10465d4606c0428250c63f7/numba-0.68.0-cp314-cp314-win_arm64.whl", hash = "sha256:f58c13a6e9bfef062311cb0d3c19f6c159b901213daa325e1db473946010cec7", size = 2812334, upload-time = "2026-09-30T15:05:24.848Z" }, + { url = "https://files.pythonhosted.org/packages/89/3b/e0e31617568553ca2b18bdf43844c44893dfb6620bde9a88296c257c5a81/numba-0.68.0-cp314-cp314t-macosx_12_0_arm64.whl", hash = "sha256:79160dc2a3ff0e02aaada2c385faa6de73d71a11f06419d29bb0a90042d243a3", size = 2763380, upload-time = "2026-09-30T15:05:27.064Z" }, + { url = "https://files.pythonhosted.org/packages/20/92/405b416800424b005c179c5b6417eee2aac1933839257ca50c855397774f/numba-0.68.0-cp314-cp314t-manylinux_2_27_aarch64.manylinux_2_28_aarch64.whl", hash = "sha256:1a3aa5558ba1c316020a0c2f6042be6ae063cfc6eb0c7badb3a0c77d2b5308b7", size = 3604721, upload-time = "2026-09-30T15:05:29.164Z" }, + { url = "https://files.pythonhosted.org/packages/e1/52/fc100dc163e12ba6a8df4c4f6e34f55d24dc6e97095f935996406d8cc946/numba-0.68.0-cp314-cp314t-manylinux_2_27_x86_64.manylinux_2_28_x86_64.whl", hash = "sha256:a08750c81fd5c2d9f2c169a73114efb907159401dde9ef4a3b629fa45e097cb7", size = 3887891, upload-time = "2026-09-30T15:05:31.234Z" }, + { url = "https://files.pythonhosted.org/packages/e1/e0/f2e074c5bf26f236c34075d390e77ed2a787c7350791b39b099b151e2033/numba-0.68.0-cp314-cp314t-win_amd64.whl", hash = "sha256:cad7d5f6fe8eb42a69c500d36c94a61d094f3b91a7a5581a31d1df2eb925d33a", size = 2838113, upload-time = "2026-09-30T15:05:33.274Z" }, + { url = "https://files.pythonhosted.org/packages/a5/85/d7cee7a6c65634bd25cb0109585785e5c8338f44db4b191c30291d9c7968/numba-0.68.0-cp315-cp315-macosx_12_0_arm64.whl", hash = "sha256:39f935bc854be87784675d9674f5503e56df5a501c95c95bdfb6b3c0b4b9ed1b", size = 2760868, upload-time = "2026-09-30T15:05:35.662Z" }, + { url = "https://files.pythonhosted.org/packages/d6/79/312e0cf6e835f700d42a223c1bd4a24b232892bded1ddf5e40bb3a329f55/numba-0.68.0-cp315-cp315-manylinux_2_27_aarch64.manylinux_2_28_aarch64.whl", hash = "sha256:7cec6809fe93824e243a8a8c93966b0bb5874a3b7c24c1194c3bafee0ab11f39", size = 3568127, upload-time = "2026-09-30T15:05:37.967Z" }, + { url = "https://files.pythonhosted.org/packages/5e/05/f31cd9e40f6d4ec6de38959e4736a917aa9d115fecc4a1979aceedcc083b/numba-0.68.0-cp315-cp315-manylinux_2_27_x86_64.manylinux_2_28_x86_64.whl", hash = "sha256:c1f1180e0332ad5143905288325485b52ac76102330811dc6f2c10088cf4cedc", size = 3853913, upload-time = "2026-09-30T15:05:40.247Z" }, + { url = "https://files.pythonhosted.org/packages/6c/28/059b2d1ea5616a5712fd722b2ec8e8278d14e4e4eb8845d36fe1658e6be8/numba-0.68.0-cp315-cp315-win_amd64.whl", hash = "sha256:a2d21bb9c4b4818a1e71721ebd19172f488591d548f08453593348b7048ba1fb", size = 2831865, upload-time = "2026-09-30T15:05:42.306Z" }, +] + [[package]] name = "numpy" version = "2.5.2" @@ -786,9 +850,10 @@ wheels = [ [[package]] name = "stamp" -version = "0.12.0" +version = "0.13.0" source = { editable = "." } dependencies = [ + { name = "fast-hdbscan" }, { name = "loguru" }, { name = "matplotlib" }, { name = "mrcfile" }, @@ -811,6 +876,7 @@ dev = [ [package.metadata] requires-dist = [ + { name = "fast-hdbscan", specifier = ">=0.3.2" }, { name = "loguru", specifier = ">=0.7.3" }, { name = "matplotlib", specifier = ">=3.11.1" }, { name = "mrcfile", specifier = ">=1.5.4" },