SeAttnFusionNet

Squeeze-excitation attention fusion for pharyngitis screening from RGB throat images, with Grad-CAM interpretability

Reference implementation for SeAttnFusionNet for pharyngitis versus no pharyngitis classification from RGB throat images with grad-CAM-based interpretability — Discover Analytics, 4(1), 22, 2026.

---

The problem

A throat photograph taken on a phone is a genuinely hostile input, and for three reasons that all push the same way. The flash blows out the near field and leaves the pharynx in shadow. The tongue occludes part of the view. Framing distance varies by a factor of three between users.

Meanwhile the diagnostic evidence — erythema, exudate, follicular pattern — occupies a small part of a frame that is mostly teeth, tongue and shadow.

A single encoder has to compromise between reading fine surface texture and reading the whole pharyngeal field. This architecture stops making that compromise.

Approach

Two encoders that fail differently.

The local encoder runs standard dilation-1 convolutions and keeps fine texture: exudate granularity, follicle edges, the surface quality of the mucosa. These are small, high-frequency and spatially local.

The context encoder runs dilated convolutions at rates 1, 2 and 4. Same parameter count, receptive field several times wider, and critically without the extra downsampling that would destroy the texture the local branch depends on. This is the branch that can answer "is the erythema symmetric across both tonsils".

Squeeze-and-excitation at three points. Inside each encoder, and again over the concatenated fusion. The third gate is the one that matters:

fused = SE(concat(local, context))

Gating after concatenation rather than before is deliberate. The useful question is not "how informative is this channel" but "how informative is this channel given what the other stream is also reporting". Gating the streams separately cannot ask that.

Grad-CAM is not an afterthought. model.cam_layer names the fusion projection explicitly, so attribution always comes from the fused representation. Attributing from the last convolution of one branch would show what that branch looked at, which is a different and misleading picture.

Design notes

• Grad-CAM implemented directly, not imported. The details libraries hide are the ones that matter: ReLU goes on the weighted sum (keeping evidence for the class), not on the gradients; normalisation is per image, never across the batch; and hooks are removed on exit, because a forward hook left attached silently retains activations for every later batch.
• Counterfactual attribution is supported. cam(images, class_index=0) explains a class the model did not predict, which is often more informative than explaining the prediction.
• Augmentation is clinically conservative. No vertical flips, no hue shifts. An upside-down throat photograph is not a plausible input, and hue is the signal here — augmenting it away trains the model to ignore erythema.
• The context branch upsamples to meet the local branch if their spatial dimensions diverge, rather than assuming they match.

Figures

_{Dataset samples. One row per class, drawn from the split on disk.}

_{Class distribution across train, validation and test, with the imbalance ratio.}

_{Architecture, generated directly from the block specification in src/model.py so it cannot drift from the code.}

_{Confusion matrix, row-normalised, from an actual evaluation run.}

About these figures. They are produced by python -m src.cli figures from the synthetic sample, not from the dataset used in the paper. They demonstrate that the pipeline runs and that the reporting is real; they are not the published results. No clinical imagery is redistributed here.

Installation

git clone https://github.com/shehan2020/seattnfusionnet.git
cd seattnfusionnet
python -m venv .venv && source .venv/bin/activate    # Windows: .venv\Scripts\activate
pip install -r requirements.txt

Python 3.10+. A GPU helps for training but nothing here requires one.

Data

No clinical imagery is redistributed. Place a dataset as:

data/raw/
  train/<class_name>/*.png
  val/<class_name>/*.png
  test/<class_name>/*.png

Or build the reproducible synthetic sample, which is what every figure above comes from:

python -m src.cli synth

Class order is derived by sorting directory names and stored in the checkpoint, so evaluation cannot silently run against a different label order — src.cli eval raises rather than reporting corrupted metrics.

Usage

Everything runs through one entry point:

python -m src.cli synth                                  # build the sample data
python -m src.cli train --config configs/default.yaml    # train
python -m src.cli eval  --checkpoint runs/latest/best.pt # evaluate
python -m src.cli figures                                # regenerate assets/

Override any config value inline:

python -m src.cli train --epochs 40 --batch-size 16 --lr 1e-4 --output runs/exp2

Report screening metrics with the positive class named:

python -m src.cli eval --checkpoint runs/latest/best.pt --positive pharyngitis

Generate Grad-CAM overlays:

import torch
from src.model import build_model
from src.gradcam import GradCAM, explain_batch

model = build_model(num_classes=2)
images = torch.randn(4, 3, 224, 224)

with GradCAM(model) as cam:
    heatmaps, explained_classes = cam(images)

# or straight to viewable overlays
overlays, classes = explain_batch(model, images)

Compare the two branches, to reproduce the fusion ablation:

local_features, context_features = model.stream_outputs(images)

Metrics

src.cli eval writes metrics.json containing accuracy, balanced accuracy, macro precision / recall / F1, weighted F1, Cohen's kappa, MCC, AUROC, the full confusion matrix and per-class figures. --positive adds sensitivity, specificity, PPV and NPV.

Checkpoints are selected on macro F1, never on accuracy.

Results

Published results for this method are in the paper cited below. This repository holds the implementation and the evaluation harness that produces such metrics; it deliberately ships no precomputed numbers, so anything reported from it is reproducible from a run you can inspect.

Layout

seattnfusionnet/
├── src/
│   ├── model.py        dual encoders, SE gates, attention fusion
│   ├── data.py         dataset, augmentation, synthetic sample generator
│   ├── engine.py       training and evaluation loops
│   ├── metrics.py      scoring, seeding, early stopping
│   ├── figures.py      everything in assets/
│   └── cli.py          single entry point
│   └── gradcam.py      Grad-CAM attribution
├── configs/default.yaml
├── assets/             generated figures
├── tests/
└── requirements.txt

Citation

@article{shehan2026seattnfusionnet,
  title   = {SeAttnFusionNet for pharyngitis versus no pharyngitis classification from RGB throat images with grad-CAM-based interpretability},
  author  = {H. M. Meghla and A. Rahim and M. S. H. Shehan and M. M. H. Melon and M. Shahiduzzaman and et al.},
  journal = {Discover Analytics, 4(1), 22},
  year    = {2026},
}

Author

Md Sazzad Hossain Shehan — Third author on this work. GitHub · LinkedIn

License

MIT — see LICENSE.

Released for research and educational use. This is not a medical device, has not been evaluated by any regulatory body, and must not be used to make clinical decisions about real patients.