Reliability analysis for BraTS-GoAT segmentation

Code and evaluation harness for Reliability analysis for BraTS-GoAT segmentation: a controlled robustness study of deep-ensemble uncertainty (MICCAI BraTS-GoAT 2026, Task 3).

The question is not only how well a model segments, but whether its own uncertainty knows when it is wrong. We compare the two uncertainty signals that come free from standard models — maximum-softmax confidence from a single network, and disagreement between deep-ensemble members — on calibration and error detection, in distribution and under graded synthetic acquisition shift.

Results

Training data. 5-fold cross-validated single nnU-Net, one held-out prediction for each of the 1351 training cases. Dice and NSD are higher-is-better; HD95 is in millimetres, lower is better. ECE is calibration error (lower is better), AUROC is error-detection ability (0.5 is chance, higher is better), AURC is the same ability expressed as a cost (lower is better).

Region	Dice	NSD	HD95 (mm)	ECE	AUROC	AURC
WT	0.925	0.858	6.45	0.0697	0.8748	0.0304
TC	0.909	0.865	6.29	0.0761	0.8716	0.0414
ET	0.869	0.891	10.73	0.0798	0.8571	0.0364

Validation data. 5-fold challenge ensemble on the official BraTS-GoAT validation leaderboard, the configuration shipped in the container. Denominators differ by region because the official pipeline leaves a region unscored where it is undefined in the reference.

Region	n scored	Dice	NSD	Lesion-wise F1	HD95 mean (mm)	HD95 median (mm)
WT	450	0.875	0.484	0.689	17.12	2.24
TC	449	0.814	0.497	0.738	25.05	2.00
ET	435	0.782	0.547	0.667	39.97	1.41

The distance between the HD95 mean and median is the point: the aggregate is set by a small tail of catastrophic cases, not by uniform degradation. Note also that the leaderboard NSD is not directly comparable with the training NSD above, because the official evaluation applies a stricter surface tolerance.

What is here

code/01_baseline/     data conversion, nnU-Net training and the seeded-ensemble trainer
code/03_evaluation/   the reliability harness: metrics, corruption protocol, statistics, figures
docker/               the container recipe used for the challenge submission (no weights)
figures/              the paper's four figures, with a .json provenance sidecar for each
results/              the aggregate numbers behind the tables above

code/03_evaluation/ is the part that carries the scientific contribution. reliability_metrics.py implements ECE, AUROC against per-voxel error, and the risk--coverage curve; seg_metrics.py handles Dice, HD95 and NSD including the empty-reference cases; corruptions.py and make_corrupted.py define the controlled robustness study; compare_paired.py runs the paired tests.

Reproducing

The scripts were written for a SLURM cluster. Set the project root once:

export GOAT_ROOT=/path/to/your/project

Every script reads that variable; the SLURM jobs fail immediately if it is unset rather than defaulting to something wrong. Update --account=your-slurm-account in the .slurm headers to your own allocation.

python code/01_baseline/convert_goat_to_nnunet.py    # GoAT -> nnU-Net layout
sbatch code/01_baseline/preprocess.slurm
sbatch code/01_baseline/train_fold0.slurm            # ~29 h per fold on one A100
sbatch code/01_baseline/train_folds1-4.slurm
sbatch code/01_baseline/train_ensemble.slurm         # 3 seeds on the fold-0 split
sbatch code/03_evaluation/eval_folds.slurm           # per-fold reliability
sbatch code/03_evaluation/stress_test.slurm          # the corruption study
python code/03_evaluation/combine_stress.py
python code/03_evaluation/eval_plots.py              # regenerates figures/

Configuration. nnU-Net v2.8.0, ResEnc-L plans, 3d_fullres, patch 160x192x160, batch 3, 1000 epochs per fold, all other settings at library defaults. No post-processing. Trained on one NVIDIA A100 per fold.

Container. docker.synapse.org/syn75822625/brats-goat:v2 (digest cea318688ac8...). Test-time mirroring is disabled in the shipped image, both in predict.py and via ENV in the Dockerfile, because the evaluation harness passes no environment overrides. Model weights are not in this repository; see docker/README.md for where the container expects them.

Data

This repository contains no imaging data and no model weights.

The BraTS-GoAT dataset is obtained from the challenge organisers through Synapse (ID syn74274097) under their terms, and is not redistributed here. results/all_scores.csv is the organiser-returned per-case score file for our own submission — evaluation metrics only, no image data.

Per the challenge rules, the models were trained only on the provided challenge data, from scratch, with no external data and no pretrained weights.

Citing

If you use this code, please cite the paper (BraTS-GoAT 2026 proceedings; full reference to follow once the volume is published) and the works the challenge requires: Karargyris et al. on MedPerf, Menze et al., Baid et al., and Bakas et al. on BraTS.

License

Apache-2.0. nnU-Net, on which this builds, is also Apache-2.0.