Sepsis Prediction — Multi-Paradigm Clinical AI

Paper: Comparing XGBoost, Temporal CNNs, and LLM Reasoning for ICU Sepsis Prediction with Interpretability Analysis arXiv submission — 2026

A reproducible research pipeline comparing three paradigms for early sepsis prediction in ICU patients, evaluated on the eICU Collaborative Research Database.

---

Overview

Paradigm	Model	AUROC	Notes
Tabular ML	XGBoost (labs+vitals)	0.919 [0.916–0.922]	Retrospective triage; temporal leakage
Tabular ML	XGBoost (vitals only)	0.704 [0.697–0.711]	Fair comparison with CNN
Timeseries DL	SepsisNet CNN	0.770 [0.764–0.775]	15-feature vitals sequences
LLM reasoning	LLM + critic	0.796 ± 0.032	Zero-shot; n=100 subset
LLM reasoning	LLM + consistency	0.777 ± 0.026	Majority vote; n=100 subset

Novel contribution: CRAS (Clinical Reasoning Alignment Score) — measures whether the LLM's reasoning focuses on the same features that XGBoost SHAP identifies as most important per patient.

CRAS = |SHAP top-3 ∩ LLM-mentioned features| / 3

Base rationale CRAS : 0.380 ± 0.221
Critic reasoning CRAS: 0.784 ± 0.019   (p < 0.0001)

---

Project Structure

sepsis-prediction/
│
├── main.py                         ← single entry point (run this)
├── config.py                       ← all paths, constants, hyperparameters
├── requirements.txt
│
├── data/
│   ├── sepsis_dataset.csv.gz       ← tabular features (XGBoost + LLM)
│   └── timeseries_clean.csv.gz     ← vitals timeseries (CNN)
│
├── models/
│   ├── xgboost_sepsis.json         ← trained XGBoost (labs+vitals)
│   ├── xgboost_vitals_only.json    ← trained XGBoost (vitals only)
│   ├── xgboost_preprocessing.pkl   ← scaler + train medians
│   └── best_sepsis_final.pt        ← SepsisNet CNN weights
│
├── outputs/
│   ├── xgboost_results.json        ← metrics + prediction arrays
│   ├── cnn_results.json            ← metrics + prediction arrays
│   ├── llm_results_seed{42,43,44}.json
│   ├── llm_aggregated.json         ← mean ± std across 3 seeds
│   ├── llm_calibration_all.csv     ← pooled reliability diagram data
│   ├── cras_detailed.csv           ← per-patient CRAS scores
│   ├── cras_summary.json
│   └── tables/
│       ├── table1_predictive.txt/.tex
│       └── table2_llm.txt/.tex
│
├── figures/
│   ├── fig1_roc.png                ← ROC curves, all models
│   ├── fig2_pr.png                 ← Precision-Recall curves
│   ├── fig3_calibration.png        ← Calibration: XGBoost + LLM
│   ├── fig4_shap_features.png      ← SHAP importance + LLM mention rate
│   ├── fig5_cras.png               ← CRAS distribution + by case type
│   ├── fig6_robustness.png         ← Consistency + ablation heatmap
│   ├── fig7_confusion_matrices.png ← Normalized CMs, all 5 models
│   ├── fig8_pipeline_gains.png     ← Full vs missing data (2×4 panel)
│   └── fig9_case_reasoning_panel.png ← Qualitative case study
│
└── src/
    ├── config.py                   ← (same as root config.py — symlink)
    ├── xgboost_save_results.py     ← evaluate XGBoost, save arrays
    ├── cnn_eval.py                 ← load CNN checkpoint, evaluate
    ├── run_llm_seed.py             ← LLM pipeline (one seed)
    ├── aggregate_seeds.py          ← combine 3 seeds → mean ± std
    ├── generate_tables.py          ← Table 1 + Table 2
    ├── unified_results.py          ← single data loader for figures
    └── figures/
        ├── fig1_roc.py ... fig9_case_reasoning_panel.py
        └── run_all_figures.py

---

Quick Start

1. Install dependencies

pip install -r requirements.txt

2. Place data and model files

data/sepsis_dataset.csv.gz       ← eICU tabular features
data/timeseries_clean.csv.gz     ← eICU timeseries
models/best_sepsis_final.pt      ← CNN weights (from training)

Data access: eICU data requires credentialing via PhysioNet. Apply at: https://physionet.org/content/eicu-crd/

3. Run the full pipeline

# Run everything (LLM steps require OpenAI API key, ~$45 total for 3 seeds)
python main.py

# Skip LLM (use saved results if available)
python main.py --skip-llm

# Regenerate figures only
python main.py --figures-only

# Run specific steps
python main.py --steps xgb cnn tables figures

# Force rerun (ignore cached outputs)
python main.py --force

4. Run on Google Colab (recommended)

LLM seeds can run in parallel across 3 notebooks to save time:

# Notebook 1 (main)
!python src/xgboost_save_results.py
!python src/cnn_eval.py
!python src/run_llm_seed.py --seed 42

# Notebook 2 (parallel)
!python src/run_llm_seed.py --seed 43

# Notebook 3 (parallel)
!python src/run_llm_seed.py --seed 44

# After all 3 seeds finish — back in Notebook 1
!python src/aggregate_seeds.py
!python src/generate_tables.py
!python src/unified_results.py
!python src/figures/run_all_figures.py

---

Reproducibility

All random seeds are fixed to SEED=42 in config.py. LLM calls use temperature=0.0 for deterministic output. Three independent seeds (42, 43, 44) are run and results reported as mean ± std.

Component	Reproducible?	Notes
XGBoost	fully	Fixed seed, saved model
CNN	fully	Fixed seed, saved checkpoint
LLM metrics	across seeds	temperature=0.0, 3-seed mean±std
LLM exact outputs	API-dependent	GPT-4.1-mini may update

---

Key Findings

1. Lab features drive XGBoost: AUROC drops from 0.919 → 0.704 removing labs. Creatinine and lactate dominate SHAP.

2. CNN temporal modeling adds value: CNN (0.770) vs XGBoost vitals-only (0.704) = +0.066 AUROC from temporal structure.

3. LLM reasoning aligns with SHAP: CRAS = 0.784 ± 0.019 — LLM discusses clinically relevant features.

4. Critic step improves reasoning quality: Base rationale CRAS 0.380 → Critic reasoning CRAS 0.784 (p < 0.0001).

5. FN cases show highest CRAS (0.933): LLM identifies correct features but still misclassifies — CRAS measures coverage, not correctness.

6. MEDIUM bucket is well-calibrated: Gap = 0.017 (predicted 0.5, actual 0.517). LOW and HIGH are miscalibrated by ~0.11.

7. Uncertainty sensitivity = 0.85: LLM predictions change in 85% of patients when lab features are removed.

---

Data & Reproducibility

The processed datasets (data/sepsis_dataset.csv.gz, data/timeseries_clean.csv.gz) and trained model weights are derived from the eICU Collaborative Research Database and cannot be redistributed under the PhysioNet Data Use Agreement.

To reproduce this work:

1. Obtain eICU access: https://physionet.org/content/eicu-crd/
2. Run preprocessing to generate the feature files
3. Place files in data/ and models/ as shown above
4. Run python main.py

All reported metrics and figures are fully reproducible from the original eICU data using the code in this repo.

---

Citation

This work is part of a graduate-level project and a research manuscript currently in preparation.

If you use this code, please cite:

@misc{mohan2026cras,
  title  = {CRAS: Benchmarking LLM Clinical Reasoning Alignment in ICU Sepsis Prediction},
  author = {Thillaikarasi Mohan},
  year   = {2026},
  note   = {Preprint in preparation for arXiv submission},
  institution = {The University of Texas at Austin}
}