Warm dark matter (WDM) N-body simulations produce artefacts called spurious haloes, unphysical objects that arise from numerical noise rather than genuine gravitational collapse (see e.g. Wang & White 2007, MNRAS 380, 93; Lovell et al. 2014, MNRAS 439, 300). They appear as regularly spaced beads along cosmic filaments and contaminate any analysis of the low-mass halo population.
The companion paper (Mostoghiu Paun et al. 2025, MNRAS 542, 735) identifies spurious haloes using an empirical sphericity cut on protohalo Lagrangian volumes, following the Lovell et al. 2014, MNRAS 439, 300 method: haloes that formed from flattened initial volumes are flagged spurious. That cut is a single threshold on a single feature, manually tuned, and shown to break down when the simulation's initial redshift changes.
This project explores the effects of replacing the empirical cut with a trained binary classifier. Labels are derived from WDM–CDM cross-correlation derived from a merit function, where a halo that exists in WDM but has no CDM counterpart is spurious by construction, decoupling the label from sphericity entirely. A follow-up SHAP analysis then asks which features actually drive classification, and whether sphericity is as informative as the paper assumes.
With ~82% of WDM haloes spurious and the label inheriting strong mass dependence below the free-streaming cutoff, aggregate metrics are nearly saturated before any learning happens: a logistic regression on log10_m200 alone reaches AP ~ 0.97–0.98 on every split. All model comparisons below are therefore read as the margin over this baseline, and each model is evaluated at its own decision threshold tuned on the validation set (max F1).
Average Precision / ROC-AUC / F1 across all models and evaluation splits:
| Model | within-sim | cross-softening | cross-z_ini |
|---|---|---|---|
| Mass-only baseline | 0.980 / 0.939 / 0.967 | 0.970 / 0.917 / 0.954 | 0.975 / 0.926 / 0.962 |
| Logistic Regression (elastic net) | 0.982 / 0.943 / 0.965 | 0.973 / 0.920 / 0.951 | 0.977 / 0.930 / 0.960 |
| Random Forest | 0.986 / 0.953 / 0.967 | 0.974 / 0.925 / 0.955 | 0.982 / 0.943 / 0.963 |
| Gradient Boosted Trees | 0.984 / 0.950 / 0.965 | 0.973 / 0.924 / 0.955 | 0.981 / 0.941 / 0.963 |
| Soft-voting Ensemble | 0.985 / 0.951 / 0.967 | 0.976 / 0.930 / 0.955 | 0.981 / 0.940 / 0.963 |
| MLP (impute) | 0.986 / 0.953 / 0.966 | 0.972 / 0.918 / 0.954 | 0.980 / 0.939 / 0.962 |
| MLP (mask) | 0.986 / 0.953 / 0.965 | 0.974 / 0.923 / 0.954 | 0.980 / 0.940 / 0.963 |
| Random Forest, no mass features | 0.982 / 0.944 / 0.965 | 0.977 / 0.930 / 0.950 | 0.978 / 0.936 / 0.961 |
Where:
within_sim: train and test on the same simulationcross_softening: train on fixed softening, test on tidal adaptive softeningcross_z_ini: train on z_ini=39, test on z_ini=99
The best model beats the mass baseline by at most ~0.007 AP in aggregate, which is exactly why the aggregate view is the wrong lens for this problem. Absolute box positions (x/y/z_norm) are deliberately not model features: all simulations share initial-condition phases, so a position-only predictor transfers across the "independent" test simulations with AUC ~ 0.7.
Three views remove the mass saturation (notebooks 05 and 07):
- ≥100-particle regime. Below 100 particles AHF properties are noise-dominated and ~97% of haloes are spurious. Above it, the class balance is nearly even (~52% spurious) and classification is a real problem. The RF holds AP ~ 0.94 / 0.91 / 0.94 there against a prevalence baseline of ~0.52.
- Mass-binned AUC. Within 0.5-dex mass bins, mass-only ranking collapses (AUC ~ 0.45–0.50 in the 8.5–9.0 bin) while the RF stays at AUC ~ 0.69–0.80 in every populated bin — including on the cross-z_ini test set. In-bin separation is genuine beyond-mass signal, and it survives both distribution shifts.
- No-mass ablation. An RF trained with every mass-family feature removed (
log10_m200,log10_npart,log10_m_hmm) loses almost nothing in aggregate (AP 0.982 / 0.977 / 0.978) and actually beats the full model on cross-softening. Kinematics (velocity dispersion), shape, and environment carry nearly all of the transferable discriminative power.
log10_m200 (with collinear log10_npart) dominates all models, as expected: spurious haloes concentrate near the WDM free-streaming mass scale, and the label inherits mass dependence through counterpart availability. v_disp_sigv (velocity dispersion) is the leading physically distinct signal beyond mass, consistently across model families — matching the ablation result above. sphericity_s, the paper's primary diagnostic, ranks mid-table at best — well below the mass features and velocity dispersion: the CDM-match label is not defined by sphericity, and ~25% of haloes lack protohalo records, diluting its importance. The paper's empirical cut is largely a mass cut in disguise; the transferable extra information lives in internal kinematics, not protohalo shape.
The primary CDM-match label is robust to both of its construction choices: requiring the match to be mutual (bijective, using the reverse CDM→WDM crossmatch) flips only ~1% of genuine labels, concentrated at the lowest masses, and the spurious fraction moves only weakly across a 15× range of merit thresholds because the best-merit distribution is strongly bimodal.
The expected structured failure on cross_z_ini did not materialise (RF AP drops just 0.004 from within-sim). The CDM-match label is robust to the initial-redshift shift because it is based on particle overlap, not morphology. Both WDM and CDM sphericities shift downward at z_ini=99, but the merit criterion is unaffected.
Counterintuitively, cross_softening degrades more (RF AP drop 0.013). Tidal adaptive softening shifts halo formation times and increases the spurious fraction near the 100-particle mass limit, creating a harder test distribution — notably, the no-mass RF handles that shift better than the full model, consistent with the mass–label relation being what shifts.
Random Forest remains the recommended model; the soft-voting ensemble adds negligible diversity.
- Pipeline: DuckDB, Polars, NumPy, SciPy
- ML: scikit-learn (mass baseline, LR, RF, GBM, Ensemble), PyTorch (MLP)
- Interpretability: SHAP
- Experiment tracking: MLflow
- Data architecture: DuckDB Databricks medallion (bronze -> silver -> gold)
- Dev tooling: ruff, basedpyright, pytest (113 tests), GitHub Actions CI
Raw files (AHF catalogues, HDF5 shapes, MergerTree cross-correlations both directions)
|
V make gold
bronze.* -- raw ingestion into 5 tables, no transforms (parsed protohalos cached as parquet)
silver.* -- column renames, best-match reduction in both directions, protohalo view
gold.* -- 12 ML features (+ position context columns), CDM-match labels
(unidirectional + bijective diagnostic), train/val/test splits
|
V make train
models/ -- mass baseline + LR/RF/GBM/ensemble + 2 PyTorch MLPs per split,
plus a no-mass RF ablation; saved as self-contained bundles
(model + feature contract + scaler stats + tuned threshold),
all tracked in MLflow
|
V notebooks/
EDA + label robustness + SHAP + generalisation analysis (01–07)
Make targets are incremental: each layer writes a stamp under data/.stamps/ and only rebuilds when config or that layer's sources change.
- Python 3.13
- Conda (recommended) or any virtual environment
git clone https://github.com/rmostoghiupaun/spurious_halo_classifier.git
cd spurious_halo_classifier
pip install -e ".[dev]"cp .env.example .env
# Set LATEX_BIN_DIR to your TeXLive bin directoryRaw simulation data (~11 GB, 8 simulations) is available on Zenodo: https://doi.org/10.5281/zenodo.20521139. Download and unpack under data/raw/ following the layout in config.yaml. The DuckDB database is built from scratch and does not need to be downloaded separately.
make gold # builds bronze -> silver -> gold in order
make train # builds any stale layers, then trains all 21 models
make lint # ruff + basedpyright
make test # pytest with coverageEach target depends on the previous layer, and stamp files under data/.stamps/
keep the chain incremental: a layer only rebuilds when config.yaml or that
layer's sources change.
The no-mass ablation (used in notebook 07) is trained separately:
python -m src.models.train_sklearn --model rf --split all --feature-set no_massTo reset and rebuild from scratch:
make reset # drop database, stamps, MLflow runs, and model artefacts
make train # rebuild everything end-to-endNotebooks 01–04 need the data pipeline (make gold); 05–07 additionally need
the trained models (make train, plus the no-mass ablation for notebook 07).
| Notebook | Description |
|---|---|
01_eda_bronze.ipynb |
Raw data: halo counts, AHF schema, protohalo coverage, merit score distribution |
02_eda_silver.ipynb |
Cleaned data: mass distributions, unit sanity checks, match statistics |
03_label_analysis.ipynb |
Label comparison and robustness: agreement, bijective (mutual-best-match) variant, merit-threshold sensitivity |
04_feature_distributions.ipynb |
Feature completeness, per-feature distributions by spurious label |
05_model_comparison.ipynb |
PR/ROC curves and confusion matrices across models and splits, all against the mass-only baseline; mass-controlled comparison |
06_shap_analysis.ipynb |
SHAP feature importance: which features drive classification across splits? |
07_generalisation.ipynb |
Why cross-z_ini did not fail; signal-beyond-mass: no-mass ablation and mass-binned AUC |
├── src/
│ ├── bronze/ # raw ingestion parsers (with parquet caching for protohalos)
│ ├── silver/ # cleaning, best-match reduction, protohalo view
│ ├── gold/ # feature contract (FEATURE_COLS/FEATURE_SETS), labels, splits
│ ├── models/ # sklearn + PyTorch training, evaluation, shared scaffolding (common.py)
│ ├── utils/ # plotting and notebook setup helpers
│ ├── config.py # config loader; parses and validates all config.yaml fields
│ └── db.py # DuckDB connection, write_table, shared utilities
├── sql/
│ ├── schema/ # DDL reference (bronze.sql, silver.sql, gold.sql)
│ └── queries/ # analytical reference queries
├── notebooks/ # EDA and analysis (01–07)
├── tests/ # 113 pytest tests across 9 modules
│ └── fixtures/ # sample AHF and MergerTree test data
├── scripts/ # MergerTree cross-correlation runner
├── reports/
│ ├── figures/ # generated plots
│ └── spurious_halo_classifier.mplstyle
├── .github/
│ └── workflows/
│ └── ci.yaml # lint, type-check, and test on every push
├── data/ # .gitkeep — database written here by make bronze
│ ├── raw/ # .gitkeep — place Zenodo data here
│ ├── interim/ # parquet cache of parsed protohalo tables (generated)
│ └── .stamps/ # Makefile incrementality stamps (generated)
├── models/ # trained model bundles: model + feature contract + threshold
├── mlruns/ # MLflow tracking store (sqlite)
├── config.yaml # all configurable parameters
├── pyproject.toml # package metadata and tool configuration
├── LICENSE # MIT 2026
├── .env.example # template for LATEX_BIN_DIR and MTREEBINPATH
└── Makefile # pipeline entrypoints
make test113 tests across 9 modules covering config validation, parsers (including the protohalo parquet cache), silver best-match transforms, gold labels (including the bijective variant) and features, model evaluation, and artefact-bundle round trips. CI runs make lint and make test on every push and pull request to main.
@article{mostoghiupaun2025,
author = {Mostoghiu Paun, R.~A. and Croton, D. and Power, C. and
Knebe, A. and Ussing, A.~J. and Duffy, A.~R.},
title = {Tidal adaptive softening and artificial fragmentation
in cosmological simulations},
journal = {MNRAS},
year = {2025},
volume = {542},
pages = {735--746},
doi = {10.1093/mnras/staf1229}
}