Disentangling Latent Risk Pathways
via Bayesian Hypergraph Inference

Unofficial Python port · maintained independently by etoyama

🌐 Original project page  ·  📄 Paper (arXiv)  ·  📌 BibTeX

---

⚠️ Disclaimer & provenance

This repository (etoyama/BHPI) is an unofficial, independent Python port of BHPI. It is not affiliated with, reviewed by, or endorsed by the paper's authors. All credit for the method, the paper, and the original implementation belongs to the original authors — Ding, Shengxian; Gao, Haonan; Liu, Pangpang; Tian, Xinyuan; Zhao, Yize; this port exists solely to make their work more accessible to Python users. Please see their paper on arXiv and the official project page for the authoritative description of the method and results, and use the BibTeX below to cite their work.

Any bugs, inaccuracies, or deviations introduced while porting this method to Python are the responsibility of the porter (etoyama) alone, not of the original authors. Please report issues about this Python package to this repository's issue tracker — do not contact the original authors about problems with this port.

The matlab/ directory holds the original authors' MATLAB implementation, kept frozen and unmodified as the reference oracle that this Python port's numerical output is validated against (see ADR-0001).

🧩 Overview

BHPI reframes multi-outcome modeling as inferring a latent hypergraph: outcomes (the paper's motivating case is diseases in electronic health records) group into overlapping pathways (hyperedges), and each risk factor acts on pathways rather than individual outcomes. An outcome's per–risk-factor effect is composed from the pathways it belongs to:

$$\beta_{j,v} = d_v^{-1} \sum_{e} H_{v,e} \mu_{j,e}$$

A repulsion prior keeps the discovered pathways parsimonious and identifiable, and a structured variational inference scheme (Pólya–Gamma augmentation + CAVI) preserves the existence → membership → effect logic for calibrated posterior uncertainty over both the outcome groupings and the risk-factor effects. Nothing about the algorithm is specific to disease data — any setting with multiple correlated binary outcomes and shared risk factors applies.

📁 Repository structure

BHPI/
├── src/bhpi/               # Python package (this port) — pip-installable public API
├── matlab/                 # MATLAB reference implementation (frozen oracle)
│   ├── BHPI.m              # core algorithm: repulsion-aware coordinate-ascent VI
│   ├── simulate_design.m   # entry point: synthetic experiments + structural recovery
│   └── helper/             # synthetic data generation, hypergraph init, repulsion utilities
└── docs/                   # PRD / architecture / design docs / ADRs

📦 Installation

PyPI publication is not yet part of this project's scope, so install from a checkout:

git clone https://github.com/etoyama/BHPI.git
cd BHPI
pip install .          # or: pip install -e .   (editable install, for development)

Default dependencies are numpy and scipy only. Python 3.11+ is required.

⚡ Quickstart

The public API is initialize → fit → predict, plus evaluation via bhpi.evaluate. This example is self-contained: it builds a small synthetic X/Y with plain numpy so it runs with no other inputs — in real use, X/Y are your own data (risk factors and binary outcomes).

import numpy as np

import bhpi
import bhpi.evaluate

rng = np.random.default_rng(0)
N, P, V, E_hat = 200, 3, 5, 2  # samples, risk factors, outcomes, max pathways

# Replace X/Y with your own data: X is (N, P) risk factors, Y is (N, V) binary outcomes.
X = rng.normal(size=(N, P))
Y = rng.integers(0, 2, size=(N, V)).astype(float)

n_train = 150
X_train, X_test = X[:n_train], X[n_train:]
Y_train, Y_test = Y[:n_train], Y[n_train:]

# 1. Initialize the variational parameters (NNMF initialization).
initials = bhpi.initialize(
    seed_init=0,
    initial_method="NNMF",
    E_hat=E_hat,
    X_train=X_train,
    Y_train=Y_train,
)

# 2. Fit the model.
model = bhpi.fit(X_train, Y_train, E_hat, initials, max_iter=50)

# 3. Predict probabilities on held-out data.
probabilities = bhpi.predict(model, X_test)

# 4. Evaluate: per-outcome AUROC.
auroc_per_outcome = bhpi.evaluate.predictive_auroc(Y_test, probabilities)
print(auroc_per_outcome)

bhpi.evaluate is imported explicitly (import bhpi.evaluate) rather than re-exported at the top level — evaluation/diagnostics are a deliberately separate, opt-in concern from the core fit→predict flow (see ADR-0017).

More advanced usage

Beyond the quickstart above, bhpi.evaluate also exposes higher-level diagnostics for users who have access to ground-truth structure or want to benchmark against a baseline — these need more inputs than the minimal quickstart, so they aren't shown above:

• bhpi.evaluate.structure_recovery — aligns a fitted model's recovered hyperedge structure (H, gamma) against known ground truth (e.g. from synthetic experiments).
• bhpi.evaluate.compare_with_baseline (with bhpi.evaluate.logistic_baseline) — compares per-outcome AUROC against an independent per-outcome logistic regression baseline.

See the docstrings in src/bhpi/evaluate.py for full signatures, or matlab/simulate_design.m for the reference usage these mirror.

🧮 MATLAB reference implementation

The following sections describe the original authors' MATLAB implementation under matlab/, reorganized here with credit to its authors (see Disclaimer & provenance). It is kept as a frozen oracle and is not required to use the Python package above.

🚀 Getting started (MATLAB)

Requirements: MATLAB (R2023a+) with the Statistics and Machine Learning Toolbox.

Reproduce the synthetic structure-recovery experiments (run from the matlab/ directory so the relative addpath("helper") resolves):

cd matlab
simulate_design

This simulates data from a known latent hypergraph, fits BHPI, and reports structural recovery (incidence H and effect γ/μ) alongside predictive AUC against the baselines.

🛠 Usage (MATLAB)

Initialize the variational parameters, fit the model, then predict and evaluate — as in matlab/simulate_design.m:

% 1. Initialize variational parameters (NNMF initialization recommended)
[initials] = cavi_initialization(seed_init, initial_method, E_hat, X_train, Y_train, []);

% 2. Fit the BHPI model
model = BHPI(X_train, Y_train, E_hat, max_iter, ...
             seed_init, initials, omega_repulsion, staged, ...
             fix_z, z_constraint, sigma2_alpha, ...
             warmup_iters, batch_size, t0, weights, tol, verbose);

% 3. Predict on held-out data
eta_val  = X_val * model.beta + model.alpha_mean;
prob_val = 1 ./ (1 + exp(-eta_val));

% 4. Score per-disease AUROC
AUROC = NaN(1, V);
for v = 1:V
    [~, ~, ~, AUROC(v)] = perfcurve(Y_val(:, v), prob_val(:, v), 1);
end
mean_auroc = mean(AUROC);

Key parameters

Argument	Meaning
E_hat	Upper bound on the number of latent hyperedges; the model self-regularizes to fewer.
omega_repulsion	Repulsion strength; > 0 disentangles redundant pathways.
initials	Starting values for the variational parameters (from cavi_initialization).
model.beta	Learned disease-specific risk-factor effects.

See the header of matlab/BHPI.m for the full argument list (staged, fix_z, warmup_iters, batch_size, …).

⏱️ Runtime & complexity

Phase	Per-iteration complexity	UK Biobank ($N \approx 277\text{K}$)
Training	$\mathcal{O}(N \cdot E \cdot (P + V))$	~74 min · ~28 GB peak
Inference	efficient matrix ops	$< 0.1$ ms / sample

Measured by the original authors on 4 × Intel Xeon 6342 cores, 60 GB RAM running the MATLAB reference implementation; inference latency is on par with logistic regression. These are not Python measurements.

Python-measured inference latency (benchmarks/bench_predict.py, not part of the MATLAB measurement above and not a CI-gated benchmark): at the reference shape N=1200, P=6, V=30, E_hat=10 (bhpi.simulate.generate's 60%-train-split stress condition, following the precedent set by benchmarks/bench_init.py / ADR-0007), bhpi.estimator.predict (one BLAS (N, P) @ (P, V) matmul) was compared against a baseline of V=30 independent sklearn.linear_model.LogisticRegression models, each called via predict_proba once — the sklearn-idiomatic way to reproduce BHPI's simultaneous multi-disease prediction. Fitting is excluded from both arms; only inference is timed, with BLAS pinned to a single thread, 3 warm-up calls, and the median of 10 measured calls used as the statistic (both arms' per-cell latency is normalized by N * V = 36,000):

Method	Per-cell median latency
bhpi.estimator.predict	0.0030 µs/cell
sklearn baseline (V × LogisticRegression.predict_proba)	0.0407 µs/cell

Ratio (BHPI / baseline): 0.074 — well under the ≤ 10 pass/fail threshold (docs/design/epic-07-packaging.md Decision: latency-threshold-rationale); BHPI's single batched matmul is markedly faster than paying sklearn's per-call overhead V times over.

Measured on: Apple M4 (10 cores), Python 3.14.0, NumPy 2.5.1, scikit-learn 1.9.0. Reproduce with uv run python benchmarks/bench_predict.py.

🗄️ Data availability

The synthetic experiments are fully reproducible from this repository; the paper's real-data results use the UK Biobank, which requires approved access and cannot be redistributed here.

✒️ Citation

If you use BHPI, please cite the original paper:

@inproceedings{ding2026bhpi,
  title     = {Disentangling Latent Risk Pathways via Bayesian Hypergraph Inference},
  author    = {Ding, Shengxian and Gao, Haonan and Liu, Pangpang and Tian, Xinyuan and Zhao, Yize},
  booktitle = {Proceedings of the 43rd International Conference on Machine Learning (ICML)},
  series    = {Proceedings of Machine Learning Research},
  publisher = {PMLR},
  year      = {2026},
  eprint    = {2606.07677},
  archivePrefix = {arXiv}
}

📜 License

Released under the MIT License, attributed to the original authors (Copyright (c) 2026 Shengxian (Naomi) Ding) — this port does not add or claim any separate copyright. See Disclaimer & provenance above.