Transformer-CNN feature fusion for image-based pharyngitis screening from throat photographs
Reference implementation for PharynFuseFormer supports image based pharyngitis screening from throat photographs using transformer CNN feature fusion — Discover Computing 29(1), 311, 2026.
A throat photograph taken on a phone is a genuinely awkward input. Illumination comes from a flash at close range, so half the image is blown out and the other half is in shadow. The tongue occludes the field. Framing distance varies by a factor of three between users. And the finding itself - erythema, exudate, follicular pattern - is a texture cue sitting inside a scene that is mostly irrelevant pixels.
CNNs and transformers fail differently on this input, which is the opening the architecture exploits.
Two branches, chosen because their failure modes do not overlap.
- The convolutional branch keeps local texture: exudate granularity, follicle edges, the surface quality of the mucosa. These cues are small, high-frequency, and spatially local - exactly what convolution is good at.
- The transformer branch models global relations: whether erythema is symmetric across both tonsils, how the affected area relates to the uvula. A CNN with a limited receptive field cannot ask that question.
Bidirectional cross-attention, not concatenation. The straightforward fusion is to pool both branches and concatenate - which produces two independent opinions and a linear layer to arbitrate. Instead each branch queries the other before pooling:
conv_tokens += attend(query=conv_tokens, key/value=trans_tokens)
trans_tokens += attend(query=trans_tokens, key/value=conv_tokens)
The two directions use separate projections. What the CNN wants to ask of global context is not the question the transformer wants to ask of local texture, so sharing weights between the directions is the wrong prior.
Gated pooling instead of mean pooling. A throat photograph is mostly irrelevant pixels. Averaging over every token dilutes the small region carrying the finding, so each branch is pooled by a learned attention score.
- Position embeddings interpolate. Users do not crop to 224px, so the transformer branch resamples its position grid rather than refusing a differently sized input.
branch_logits()exists for the ablation table. Each branch can be scored alone, which is how the contribution of fusion was measured.- Mild augmentation only. No vertical flips - an inverted throat photograph is not a plausible input and training on one wastes capacity.
Input (3, 224, 224)
│
├──────────────┬──────────────────┐
│ │ │
CNN branch Transformer branch │
inverted patch 16x16 │
residual + interpolable pos │
blocks 6 encoder layers │
-> 256ch -> 256ch │
│ │ │
└──────┬───────┘ │
│ │
Cross-attention fusion ×2 │
conv <- queries - transformer │
transformer <- queries - conv │
│ │
┌──────┴───────┐ │
Gated pool Gated pool │
(conv) (transformer) │
└──────┬───────┘ │
│ │
concat -> LayerNorm -> MLP -> num_classes
7.4M parameters
git clone https://github.com/abedur/pharynfuseformer.git
cd pharynfuseformer
python -m venv .venv && source .venv/bin/activate # Windows: .venv\Scripts\activate
pip install -r requirements.txtRequires Python 3.10+ and PyTorch 2.1+. A GPU is recommended for training but not required — every module in this repository runs on CPU.
No clinical imagery is redistributed here. Place a dataset in this layout:
data/
train/<class_name>/*.png
val/<class_name>/*.png
test/<class_name>/*.png
Class order is derived by sorting directory names and is stored in the
checkpoint, so evaluation cannot silently run against a different label order —
src/evaluate.py raises rather than reporting corrupted metrics.
Verify the model builds and the shapes are right:
python src/model.pyTrain:
python -m src.train --config configs/default.yaml --data-root data/Evaluate a checkpoint on the held-out test split:
python -m src.evaluate --checkpoint runs/latest/best.pth --data-root data/Override any config value from the command line:
python -m src.train --epochs 40 --batch-size 12 --lr 1e-4 --output runs/exp2Score each branch separately to reproduce the fusion ablation:
conv_features, trans_features = model.branch_logits(images)Reported results for this method are in the published paper cited below. This repository contains the implementation and the evaluation harness that produces those metrics; it does not ship precomputed numbers, so that anything reported from it is reproducible from a run you can inspect.
src/evaluate.py writes a metrics.json containing accuracy, macro precision /
recall / F1, weighted F1, Cohen's kappa, AUROC, the full confusion matrix, and
per-class figures. Pass --positive-class <name> to add sensitivity,
specificity, PPV and NPV.
pharynfuseformer/
├── src/
│ ├── model.py dual branches and cross-attention fusion
│ ├── dataset.py dataset, transforms, class weighting
│ ├── train.py training loop, early stopping, checkpointing
│ ├── evaluate.py held-out evaluation and metric export
│ └── utils.py seeding, metrics, latency and parameter accounting
├── configs/
│ └── default.yaml the configuration used for the reported runs
├── tests/ shape and invariant checks
└── requirements.txt
@article{rahman2026pharynfuseformer,
title = {PharynFuseFormer supports image based pharyngitis screening from throat photographs using transformer CNN feature fusion},
author = {M. A. Hossain and M. A. Rahman and M. S. Hossain and C. Karmakar and M. S. Rahman and et al.},
journal = {Discover Computing 29(1)},
year = {2026}
}Md Abedur Rahman — Second author on this work. GitHub · ORCID · LinkedIn
MIT — see LICENSE.
Released for research and educational use. This is not a medical device and has not been evaluated by any regulatory body. It must not be used to make clinical decisions about real patients.