Skip to content

About

UMAP Neural ODE for manifold drift analysis

Resources

Stars

7 stars

Watchers

0 watching

Forks

Latest commit

 

History

1 Commit

Folders and files

Repository files navigation

Neural ODE Forecasting of Real UMAP Embeddings

Trains a Neural Ordinary Differential Equation on real text embeddings and forecasts how their UMAP layout drifts over time.

The current pipeline:

  1. Download AG News from Hugging Face.
  2. Encode texts with sentence-transformers/all-MiniLM-L6-v2.
  3. Compress embeddings with PCA (default 12D latent).
  4. Build smooth temporal snapshots (280 timesteps, 800 points each).
  5. Train a Neural ODE (RK4 solver + geometry regularizer) on the latent trajectories.
  6. Project forecasts to 2D with a fixed UMAP transformer for visualization.
  7. Quantitatively compare predictions to held-out future snapshots (Chamfer, centroid drift, covariance deviation, RMSE).

Installation

Use a fresh virtualenv if possible:

python3 -m venv .venv
source .venv/bin/activate
pip install --upgrade pip
pip install -r requirements.txt

Requires Internet access for Hugging Face downloads.

Legacy scripts (umap_neural_ode_example.py, umap_neural_ode_example_v2.py, umap_neural_ode_example_v3.py) are preserved under older_versions/ for reference. The current entry point composes modular code under src/ (data_pipeline.py, modeling.py, visualization.py).

Usage

python run_umap_neural_ode.py

Key parameters live near the top of main():

  • HOLDOUT_STEPS: number of final timesteps reserved for validation (default 40)
  • TRAIN_EPOCHS: ODE training epochs (default 300)
  • EXTRA_FORECAST_STEPS: how far beyond holdout we predict (default 80)
  • points_per_label, n_timesteps, latent_dim: set inside build_textual_snapshots

Expect >30 minutes on CPU (embedding + training). GPU shortens the run drastically.

Outputs

All artifacts land in the repo root:

  • artifacts/neural_ode_results_v3.png – six-panel plot (UMAP history, forecasts, holdout overlay, tracked trajectories, per-time RMSE, reconstruction error).
  • artifacts/actual_vs_predicted_drift_v3.png – three-panel comparison (actual holdout, predicted, overlay with shared color gradient).
  • artifacts/neural_ode_animation_v3.gif – animation that transitions from training snapshots to forecast frames, overlaying actual vs predicted during the holdout horizon.
  • Console metrics summarizing UMAP RMSE, Chamfer distance, centroid drift, covariance deviation, and latent-space errors.

Example metric block:

======================================================================
Holdout UMAP space error (mean ± std): 1.27 ± 1.19
Holdout UMAP space error (median): 0.87
Holdout UMAP space error (max): 7.15
Chamfer distance (UMAP) mean ± std: 0.24 ± 0.04 (median 0.24)
Centroid drift error mean ± std: 0.35 ± 0.16 (median 0.41)
Covariance deviation mean ± std: 0.92 ± 0.37 (median 1.05)

Holdout latent-space error (mean): 0.31
Holdout latent-space error (median): 0.32
Final training loss: 0.11
======================================================================

How It Works

  1. Data ingestion: For each AG News label, we keep 200 reference texts and an extra buffer (~320) to drive drift. We encode everything once with the SentenceTransformer model and project into a 12D PCA latent.
  2. Temporal evolution: Each label's centroid follows the latent trajectory of future texts and undergoes small rotations/noise, creating smooth manifolds while preserving document identity.
  3. Neural ODE: A velocity network takes (latent, t) and outputs dx/dt. Training uses RK4 integration and a loss = MSE + geometry regularizer (encourages local kNN distances to match).
  4. Forecasting: After training, we integrate forward from the last observed snapshot across the holdout horizon plus extra steps.
  5. Visualization: Predicted latents are passed through the fixed UMAP transformer for plotting. Histories, forecasts, and holdouts share the same 2D basis.
  6. Metrics: Besides RMSE, we compute Chamfer distance (set-level similarity), centroid drift, covariance deviation, and latent-space errors for every holdout timestep.

Customizing the Pipeline

  • Different data: Replace fetch_ag_news_texts with your own loader (news feeds, blogs, tweets). Keep consistent identities if you want point-level trajectories; otherwise swap the loss with a set/distribution metric.
  • Embedding model: Change HF_MODEL_NAME to any SentenceTransformer (or your local encoder). Update latent_dim accordingly.
  • Geometry regularizer: Tune geometry_weight, geometry_sample, geometry_k to balance fidelity vs. runtime. Set to 0 to disable.
  • Solver: NeuralODEPredictor supports rk4 (default) and euler. You can extend it with adaptive methods (e.g., torchdiffeq).
  • Prediction horizon: Adjust HOLDOUT_STEPS and EXTRA_FORECAST_STEPS. Forecast accuracy decays beyond the holdout length; re-train as new data arrives.

Limitations

  • Points must have consistent identities across timesteps. For domains with churn (points added/removed), switch to a distributional loss (Chamfer/Sinkhorn) or track a stable subset.
  • UMAP is used only for visualization; the true dynamics happen in latent space. Large UMAP distortions can still affect the 2D visual match.
  • Runtime is dominated by embedding + training. Use GPU or cache embeddings if you iterate frequently.
  • Predictions extrapolate existing dynamics. Sudden regime shifts or novel topics require retraining.

Sharing & Next Steps

This repo now contains a reproducible example of forecasting real manifold drift. Before publishing, ensure the artifacts above exist and consider caching the AG News embeddings for users without network access. Potential extensions:

  • CLI flags / config files for dataset/model choices.
  • Distribution-aware training when identities change.
  • Alternate metrics (Sinkhorn distance, persistence images).
  • Support for stochastic dynamics (SDEs) to handle multi-modal futures.

For theoretical background, UMAP’s role, caveats, and future directions see THEORY.md.

About

UMAP Neural ODE for manifold drift analysis

Resources

Stars

7 stars

Watchers

0 watching

Forks

Releases

Packages

Contributors

Languages