Trains a Neural Ordinary Differential Equation on real text embeddings and forecasts how their UMAP layout drifts over time.
The current pipeline:
- Download AG News from Hugging Face.
- Encode texts with
sentence-transformers/all-MiniLM-L6-v2. - Compress embeddings with PCA (default 12D latent).
- Build smooth temporal snapshots (280 timesteps, 800 points each).
- Train a Neural ODE (RK4 solver + geometry regularizer) on the latent trajectories.
- Project forecasts to 2D with a fixed UMAP transformer for visualization.
- Quantitatively compare predictions to held-out future snapshots (Chamfer, centroid drift, covariance deviation, RMSE).
Use a fresh virtualenv if possible:
python3 -m venv .venv
source .venv/bin/activate
pip install --upgrade pip
pip install -r requirements.txtRequires Internet access for Hugging Face downloads.
Legacy scripts (umap_neural_ode_example.py, umap_neural_ode_example_v2.py, umap_neural_ode_example_v3.py) are preserved under older_versions/ for reference. The current entry point composes modular code under src/ (data_pipeline.py, modeling.py, visualization.py).
python run_umap_neural_ode.pyKey parameters live near the top of main():
HOLDOUT_STEPS: number of final timesteps reserved for validation (default 40)TRAIN_EPOCHS: ODE training epochs (default 300)EXTRA_FORECAST_STEPS: how far beyond holdout we predict (default 80)points_per_label,n_timesteps,latent_dim: set insidebuild_textual_snapshots
Expect >30 minutes on CPU (embedding + training). GPU shortens the run drastically.
All artifacts land in the repo root:
artifacts/neural_ode_results_v3.png– six-panel plot (UMAP history, forecasts, holdout overlay, tracked trajectories, per-time RMSE, reconstruction error).artifacts/actual_vs_predicted_drift_v3.png– three-panel comparison (actual holdout, predicted, overlay with shared color gradient).artifacts/neural_ode_animation_v3.gif– animation that transitions from training snapshots to forecast frames, overlaying actual vs predicted during the holdout horizon.- Console metrics summarizing UMAP RMSE, Chamfer distance, centroid drift, covariance deviation, and latent-space errors.
Example metric block:
======================================================================
Holdout UMAP space error (mean ± std): 1.27 ± 1.19
Holdout UMAP space error (median): 0.87
Holdout UMAP space error (max): 7.15
Chamfer distance (UMAP) mean ± std: 0.24 ± 0.04 (median 0.24)
Centroid drift error mean ± std: 0.35 ± 0.16 (median 0.41)
Covariance deviation mean ± std: 0.92 ± 0.37 (median 1.05)
Holdout latent-space error (mean): 0.31
Holdout latent-space error (median): 0.32
Final training loss: 0.11
======================================================================
- Data ingestion: For each AG News label, we keep 200 reference texts and an extra buffer (~320) to drive drift. We encode everything once with the SentenceTransformer model and project into a 12D PCA latent.
- Temporal evolution: Each label's centroid follows the latent trajectory of future texts and undergoes small rotations/noise, creating smooth manifolds while preserving document identity.
- Neural ODE: A velocity network takes
(latent, t)and outputsdx/dt. Training uses RK4 integration and a loss = MSE + geometry regularizer (encourages local kNN distances to match). - Forecasting: After training, we integrate forward from the last observed snapshot across the holdout horizon plus extra steps.
- Visualization: Predicted latents are passed through the fixed UMAP transformer for plotting. Histories, forecasts, and holdouts share the same 2D basis.
- Metrics: Besides RMSE, we compute Chamfer distance (set-level similarity), centroid drift, covariance deviation, and latent-space errors for every holdout timestep.
- Different data: Replace
fetch_ag_news_textswith your own loader (news feeds, blogs, tweets). Keep consistent identities if you want point-level trajectories; otherwise swap the loss with a set/distribution metric. - Embedding model: Change
HF_MODEL_NAMEto any SentenceTransformer (or your local encoder). Updatelatent_dimaccordingly. - Geometry regularizer: Tune
geometry_weight,geometry_sample,geometry_kto balance fidelity vs. runtime. Set to0to disable. - Solver:
NeuralODEPredictorsupportsrk4(default) andeuler. You can extend it with adaptive methods (e.g.,torchdiffeq). - Prediction horizon: Adjust
HOLDOUT_STEPSandEXTRA_FORECAST_STEPS. Forecast accuracy decays beyond the holdout length; re-train as new data arrives.
- Points must have consistent identities across timesteps. For domains with churn (points added/removed), switch to a distributional loss (Chamfer/Sinkhorn) or track a stable subset.
- UMAP is used only for visualization; the true dynamics happen in latent space. Large UMAP distortions can still affect the 2D visual match.
- Runtime is dominated by embedding + training. Use GPU or cache embeddings if you iterate frequently.
- Predictions extrapolate existing dynamics. Sudden regime shifts or novel topics require retraining.
This repo now contains a reproducible example of forecasting real manifold drift. Before publishing, ensure the artifacts above exist and consider caching the AG News embeddings for users without network access. Potential extensions:
- CLI flags / config files for dataset/model choices.
- Distribution-aware training when identities change.
- Alternate metrics (Sinkhorn distance, persistence images).
- Support for stochastic dynamics (SDEs) to handle multi-modal futures.
For theoretical background, UMAP’s role, caveats, and future directions see THEORY.md.