RSNA Knee MRI Abnormality Detection

This project is my solution to the RSNA Knee Abnormality Detection competition.

The task is to build a model that looks at a knee MRI exam and predicts twelve possible abnormalities such as an ACL tear, meniscus damage, or fluid buildup (effusion). For each exam, the model outputs twelve probabilities, one per condition.

The data has a hierarchy

One knee exam is called a study. A study is made up of several series -- separate scans of the same knee, each acquired with different settings. Each series is a stack of 2D slices which are individual images.

So, we have:

study -> several series -> each series is a stack of slices

Why series are kept separate instead of mixed together

Series differ in two important ways: the plane (sagittal, coronal, or axial, basically, which direction the scanner sliced through the knee) and the sequence weighting, a scanner setting that changes what shows up bright or dark in the image. Fluid-sensitive sequences make fluid and inflammation brighter. Structural (T1) sequences show anatomy and shape more clearly.

This is important because different abnormalities are easiest to see in different series. For example, an ACL tear is best seen on sagittal, fluid-sensitive views -- the slots we call SAG_FLUID_FS and SAG_FLUID_NOFS. A structural T1 series is much less useful for diagnosing that same tear.

Because of this, the pipeline doesn't mix every series into one combined input. It keeps a small number of meaningful series types, called slots, separate, so the model can later decide which slot matters for which diagnosis. Not every study has every slot, and the model needs to know that.

From raw scan to model input

Before a series reaches the model, each one goes through the same steps:

1. Order the slices by where they actually sit in the body, using the 3D coordinates stored in each slice's DICOM data, not by filename. Filenames don't reliably match physical order, but every slice carries its real (x, y, z) position in millimeters, so sorting by that position guarantees the stack moves smoothly through the knee.
2. Sample slices from the middle of the stack, since the clinically relevant anatomy tends to sit in the central region rather than at the edges.
3. Crop to a fixed physical size in millimeters, using each scan's own pixel spacing, so a crop represents the same real-world area no matter which scanner produced the image.
4. Resize to one fixed pixel resolution, since native image sizes vary a lot across the dataset.
5. Normalize left/right, so the model always sees a knee presented the same way, regardless of which physical knee it was.

From image to prediction

Each processed slice goes into DINOv2, a pretrained image encoder, which turns an image into an embedding. Slices within a series get combined into one embedding per slot.

A study now has up to six slot embeddings, some possibly missing. For each of the twelve diagnoses separately, an attention mechanism looks across the available slot embeddings and learns how much weight to give each one for that specific diagnosis. For effusion it might rely on the fluid-sensitive slots, for a fracture it might turn to structural slots instead. Missing slots are masked out.

As a result we get 12 abnormality scores.

What EDA established

Before writing any of this as code, we checked the real data to make sure the plan above actually fits it:

• The public metadata columns alone don't reliably describe series settings in this dataset — the real DICOM headers do, and sequence type has to be recovered from those headers.
• A 130mm physical crop works across almost the entire dataset.
• Native image sizes vary enormously, so resizing after cropping is crucial.
• About 7% of images are rectangular rather than square, so preprocessing can't assume a square input.
• The laterality (left/right) tag is missing for roughly half of all studies, and this varies by scanner manufacturer rather than randomly, so we'll use a method, based on where the image sits in 3D space.