Latent Liquidity Quant

Bayesian systemic stress monitoring with a Differentiable Particle Filter and Sinkhorn OT resampling.

🌐 Interactive Portfolio • 📊 Research Visualizations • 📄 Read Paper

Detects endogenous financial crises up to ~2 months in advance from market microstructure signal.
Achieves ROC-AUC 0.68 and significantly improves regime separation vs a standard particle filter.
Designed for real-time systemic risk monitoring and early-warning applications.

---

What This Project Does

Markets do not directly reveal systemic stress.
Instead, stress manifests indirectly through volatility, spreads, and correlation structure.

This project infers that hidden stress in real time using a Bayesian state-space model — recovering a latent liquidity stress factor $L_t$, log-volatility $h_t$, and correlation driver $z_t$ from six daily observables (SPY/TLT/HYG returns, VIX, HY credit spread, rolling correlation).

---

Why This Matters

Systemic crises are driven by endogenous liquidity breakdowns — not just external shocks.

This model:

• Identifies hidden stress buildup before crises materialize
• Distinguishes endogenous vs exogenous events
• Provides actionable early-warning signals for risk management and macro trading

Two particle filter architectures are evaluated head-to-head:

	Bootstrap PF (BPF)	Differentiable PF (DPF)
Parameters	Fixed	Online gradient learning
Resampling	Multinomial	Sinkhorn OT (differentiable)
Particles	1 000	500
Early-warning AUC (endogenous)	0.500	0.567
Crisis/calm discrimination	54×	340×

---

---

Key Results

The DPF outperforms the BPF on every primary warning metric.

Metric	DPF	BPF
Daily ROC-AUC	0.682	0.670
PR-AUC	0.102	0.097
Brier Score	0.212	0.218
Endogenous Episode AUC	0.567	0.500
Crisis/calm ratio	340×	54×

5 of 9 crisis episodes detected with 13–60 day advance warning.
3 exogenous shocks (COVID-19, SVB, Rate Shock) correctly produce no signal — financial microstructure contains no 30-day precursor for external catalysts.

Per-crisis lead times:

Crisis	DPF	BPF	Type
GFC (2007–2009)	13d	26d	Endogenous
Eurozone I (2010)	58d	58d	Endogenous
Eurozone II (2011)	56d	60d	Endogenous
China (2015)	60d	60d	Endogenous
Brexit (2016)	60d	60d	Exogenous
Q4 Selloff (2018)	—	—	Endogenous
COVID-19 (2020)	—	—	Exogenous
Rate Shock (2022)	—	—	Endogenous
SVB (2023)	37d	37d	Exogenous

---

Why DPF Works Better

The Bootstrap PF uses fixed parameters and cannot adapt to changing market regimes.

The DPF:

• Learns parameters online via gradient updates
• Adapts to evolving volatility and correlation structure
• Produces more responsive and higher signal-to-noise stress estimates

Result: better crisis regime detection and cleaner separation of stress vs calm periods.

DPF improves signal quality and regime discrimination at a small cost in lead time and tail sharpness relative to the bootstrap filter.

---

Limitations

The model does not predict purely exogenous shocks (e.g. COVID-19), as these lack observable microstructure precursors.

---

Model Architecture

Observables (6):        r_SPY  r_TLT  r_HYG  VIX  Spread  Corr
                              ↓ Bayesian filter ↓
Latent states (3):        L_t       h_t       z_t
                     [stress]  [log-vol]  [correlation]

Transition model with two structural improvements:

• Stochastic Volatility with Leverage (SVL) — $h_t$ transition uses the standardised previous return, capturing the empirical fact that volatility rises when prices fall ($\rho_\text{lev} \approx -0.73$)
• Diagonal-Rotation-Diagonal (DRD) covariance — each asset gets its own baseline volatility ($\sigma_\text{SPY}=1%$, $\sigma_\text{TLT}=0.8%$, $\sigma_\text{HYG}=0.6%$/day) scaled by the common regime $e^{h_t/2}$, replacing the restrictive equal-variance assumption

DPF online learning — rolling 120-day windows, Adam with persistent state, gradient clipping, NaN rollback, and loss spike detection. 10 parameters learned jointly including the leverage correlation.

Early-warning layer — logistic regression on three rolling z-scored features: stress level, 5-day stress momentum, and equity drawdown depth. Calibrated on 30-day pre-crisis labels with COVID and SVB held out entirely.

Probability rescaling — forward simulation outputs are rescaled relative to the model's neutral baseline probability, yielding clean [0,1] crisis probabilities rather than the saturated values produced by naive threshold aggregation.

---

Quickstart

# Install
pip install -r requirements.txt

# Download and process data
python data_pipeline/download.py
python data_pipeline/build_dataset.py

# Run the full pipeline end-to-end
python main.py run

python main.py run executes all eight steps automatically:

Step 1 / 8  —  Online DPF Parameter Training
Step 2 / 8  —  DPF Prediction  (first pass — heuristic scoring)
               ↳ gate check: NaNs=0, Max L_t < 8
Step 3 / 8  —  DPF Calibration
Step 4 / 8  —  DPF Prediction  (second pass — calibrated scoring)
Step 5 / 8  —  BPF Prediction  (first pass — heuristic scoring)
               ↳ gate check: NaNs=0, Max L_t < 8
Step 6 / 8  —  BPF Calibration
Step 7 / 8  —  BPF Prediction  (second pass — calibrated scoring)
Step 8 / 8  —  DPF vs BPF Benchmark  →  results/plots/comparison/

The double predict-calibrate-predict pattern is intentional: calibration trains on the first-pass posterior states; the second pass re-scores forward simulations using the fitted logistic model.

---

Individual Commands

# Train DPF online (rolling 120-day windows, Adam, gradient clipping)
python main.py train

# Run a single filter
python main.py predict --filter dpf
python main.py predict --filter bpf

# Calibrate warning model
python main.py calibrate --filter dpf
python main.py calibrate --filter bpf

# Full metric comparison + 6 publication figures
python main.py bench

# Animated dual-filter GIF for any date window
python main.py animate --start 2007-06-01 --end 2009-06-01
python main.py animate --start 2019-09-01 --end 2021-01-01

---

Repository Structure

latent_liquidity_quant/
│
├── main.py                         # Unified CLI (train / predict / calibrate / bench / animate)
├── requirements.txt
│
├── src/
│   ├── models/
│   │   ├── finance_state_space.py  # NumPy BPF — fixed params, SVL, DRD covariance
│   │   └── tf_finance_ssm.py       # TensorFlow DPF — same model, differentiable
│   ├── filters/
│   │   ├── diff_particle_filter.py # Bootstrap model wrapper
│   │   ├── dpf_resampling.py       # Sinkhorn OT resampling
│   │   └── sinkhorn.py             # Sinkhorn iteration (ε=0.5, 20 iters)
│   └── core/
│       └── crisis_labels.py        # 30-day pre-crisis labels + holdout logic
│
├── research/
│   ├── train_dpf_online.py         # Rolling-window online parameter learning
│   ├── dpf_crisis_predictor.py     # DPF filter + forward simulation
│   ├── crisis_predictor.py         # BPF filter + forward simulation
│   ├── calibrate_crisis_score.py   # Logistic early-warning calibration
│   ├── compare_filters.py          # All metrics + 6 publication figures
│   ├── animate_comparison.py       # Dual-filter animated GIF
│   └── validate_calibration.py     # Reliability diagrams + Brier decomposition
│
├── data_pipeline/
│   ├── download.py                 # Yahoo Finance + FRED data download
│   ├── build_dataset.py            # Feature engineering + NPY export
│   └── features.py                 # Rolling correlation, drawdown, spread processing
│
├── data/
│   ├── processed/                  # observations.npy, dates.npy (model-ready)
│   └── raw/                        # financial_data.csv
│
├── results/
│   ├── trained_params/
│   │   └── dpf_learned_params.npz  # Full parameter trajectory (10 params × 74 windows)
│   ├── dpf_crisis_res.npz          # DPF posteriors + crisis probabilities
│   ├── crisis_res.npz              # BPF posteriors + crisis probabilities
│   ├── crisis_calibration_dpf.pkl  # Fitted logistic pipeline (DPF)
│   ├── crisis_calibration_bootstrap.pkl
│   └── plots/
│       ├── comparison/             # figure_1 through figure_6 + GFC/COVID GIFs
│       ├── dpf/                    # DPF-specific diagnostics + animations
│       └── bootstrap/              # BPF reliability diagrams
│
├── paper/
│   └── latent_liquidity_quant.pdf           # Full technical write-up (15 pages)
│
├── tests/
│   ├── test_model.py               # State-space transition + likelihood
│   ├── test_filters.py             # BPF/DPF correctness
│   ├── test_state_space.py         # SVL leverage, DRD covariance
│   └── test_logic.py               # Crisis labels, calibration pipeline
│
└── notebooks/
    ├── data_visualization.ipynb
    └── exploratory_analysis.ipynb

---

Outputs

After python main.py run, the key files are:

File	Description
results/plots/comparison/figure_5_performance_dashboard.png	Summary dashboard — KPIs, crisis prob timeline, lead-time bars
results/plots/comparison/figure_6_crisis_type_breakdown.png	Endogenous vs exogenous discrimination
results/plots/comparison/figure_1_stress_comparison.png	$L_t$ time series, both filters, 2007–2026
results/plots/comparison/gfc.gif	Animated dual-filter forecast cones, GFC window
results/plots/comparison/covid.gif	Animated dual-filter forecast cones, COVID window
results/trained_params/dpf_learned_params.npz	Full parameter trajectory across all 74 training windows
paper/research_note.pdf	Technical write-up with model derivations and results tables

---

Technical Highlights

Why Sinkhorn OT over multinomial resampling?
Multinomial resampling introduces a discrete stochastic step that blocks gradient flow. Sinkhorn OT replaces it with a smooth, differentiable transport plan — enabling end-to-end gradient-based parameter learning through the particle filter.

Why the predict-calibrate-predict double pass?
The calibration model (logistic regression) is trained on posterior states from the first pass. The second pass uses the fitted model to score forward simulation paths, ensuring no circularity between what the logistic model was trained on and what it evaluates.

Why a separate fixed-parameter SSM for DPF forward simulation?
The DPF's adaptive parameters are fit to the most recent 120-day window. Using them for 60-day forward simulation amplifies transient regime-specific noise, inflating calm-period crisis probabilities. The fixed-parameter SSM provides stable long-run dynamics for simulation while the DPF's adaptive filtering still drives the posterior state distribution.

Why rolling z-score features for the DPF?
As the DPF's parameters adapt, the absolute scale of $L_t$ shifts across parameter epochs. A single logistic model cannot learn a stable decision boundary from a non-stationary series. Rolling 252-day z-scoring renders $L_t$ and its 5-day momentum stationary and epoch-comparable.

---

Tests

pytest tests/ -v

Tests cover the state-space transition equations, SVL leverage direction, DRD positive-definiteness, filter weight normalisation, crisis label construction, and calibration pipeline consistency.

---

Dependencies

numpy  pandas  scipy  scikit-learn
tensorflow  joblib
matplotlib  tqdm
yfinance  pandas-datareader

Python 3.9+.
TensorFlow 2.x (CPU sufficient for training; GPU not required).

---

Citation / Reference

Full derivations, algorithm pseudocode, and results tables are in paper/research_note.pdf.

If you use this work, please cite:

Verma, A. (2025). Latent Liquidity Quantification: A Differentiable 
Particle Filter for Systemic Stress Monitoring and Crisis Early Warning.
github.com/meamresh/latent_liquidity_quant

---

License

MIT — see LICENSE.