CS 131 final project (Stanford, Spring 2026) — Brooke Ballhaus, ballhaus@stanford.edu.
Pipeline: scrape 1.50 m showjumping broadcasts from YouTube → cut takeoff-anchored clips around each detected jump → cache YOLOv8 horse-box tracks and rank clips by takeoff lift → hand-annotate outcomes and fence boxes → recover takeoff distance d from pole-length geometry → self-supervised pretrain an R(2+1)D-18 video encoder (InfoNCE + temporal-order + a domain-adversarial venue head, on a venue-balanced sampler) → fine-tune small heads for outcome classification and d regression → benchmark against three baselines and measure the venue shortcut with a leave-one-venue-out test.
The central problem the project fights is the venue confound: clips from the same broadcast share an arena and backdrop, so a naive encoder learns the venue, not the jump. The final pipeline addresses this with a balanced sampler, a domain-adversarial head, and honest cross-venue evaluation.
showjumping-ssl/
├── src/
│ ├── data/ # YouTube scrape (+ clip_sources.txt), takeoff-anchored + fixed segmentation,
│ │ # YOLO presence filter, knockdown miner, venue map, label-table builder,
│ │ # offline tuners (cache_scan, jumps_local)
│ ├── preprocess/ # YOLO horse + fine-tuned fence detector, fence-type heuristic + CNN,
│ │ # takeoff detection, geometric d, track caching, Colab/OpenCV annotators
│ ├── ssl/ # R(2+1)D-18 encoder, dataset + venue-balanced sampler,
│ │ # contrastive + order losses, domain-adversarial head, training loop, embed
│ ├── downstream/ # labeled-clip dataset, outcome + d heads, splits, training/eval loop
│ ├── baselines/ # from-scratch supervised, ImageNet 2D-CNN, type-only logistic regression
│ ├── eval/ # metrics, sample-efficiency sweep, ablations, cross-venue, venue probe
│ └── viz/ # t-SNE, d histogram, training/efficiency curves, confusion, detection grids
├── notebooks/
│ ├── colab_final.ipynb # final pipeline (venue fix, downstream, baselines, analysis)
│ ├── takeoff_validation.ipynb # validate/tune the takeoff detector; offline iteration caches
│ └── colab_milestone.ipynb # earlier milestone stage (scrape -> SSL -> milestone figures)
├── tests/
│ └── test_smoke.py # CPU-only smoke tests (no network, GPU, or video)
├── .github/workflows/tests.yml # CI: runs the smoke tests on push / PR to main
├── data/
│ ├── annotations/ # committed
│ │ ├── fences.csv # hand-drawn fence boxes (clip_id,x1,y1,x2,y2,pole_count)
│ │ ├── takeoffs.csv # hand-marked ground-truth takeoff frames (clip_id,takeoff_frame)
│ │ ├── tracks.json # cached YOLO horse-box trajectories (input to offline tuning)
│ │ ├── auto.csv # YOLO + geometry table (type, d_meters, mpp, takeoff_frame, ...)
│ │ ├── auto_clean.csv # auto.csv variant (untyped clips marked 'unannotated'), reference table
│ │ └── by_video.csv # clip_id -> source video map
│ └── results/ # committed: ablations.csv, sample_efficiency.csv, train_log.csv,
│ # takeoff_local/signals.png, jumps_local/velocity*.png
├── final_deliverables/ # submitted proposal, milestone, final report PDFs + demo-day .pptx
├── requirements.txt # torch/torchvision intentionally omitted (see comment inside)
├── AGENTS.md, CITATION.cff, LICENSE
│
│ # Created by the pipeline, gitignored, not in a fresh clone:
├── data/raw/, data/clips/ # downloaded YouTube videos and takeoff-anchored clips
├── data/annotations/outcomes.csv # hand-labeled outcomes, produced by the annotator
├── data/annotations/labels.csv # merged downstream label table, built by src.data.labels
├── data/embeddings.npz, data/fence_yolo/
├── checkpoints/, runs/ # SSL encoder, fence YOLO, fence-type CNN weights; run outputs
└── milestone/figures/ # default output dir of src.viz.make_figures
Anything large or regenerable (raw videos, clips, checkpoints, embeddings, generated figures, outcomes.csv/labels.csv) is in .gitignore; source, hand annotations, cached tracks, pipeline reference tables, the result CSVs/plots in data/results/, and the submitted deliverables in final_deliverables/ live in the repo.
The end-to-end path is notebooks/colab_final.ipynb. Open it in Colab, set GITHUB_REPO and DRIVE_DATA_ROOT to your fork in the Setup cell, and run top to bottom. Cell 0 clones this repo into /content/project and symlinks data/ + checkpoints/ to Drive, so code stays fresh from GitHub while data and weights persist across sessions. Each numbered section saves its intermediate outputs to Drive.
Sections, in order: segmentation → outcome annotation → knockdown targeting → fence annotation → fence YOLO + type CNN → YOLO horse + geometric d → label merge → SSL pretraining → embeddings → downstream heads → baselines → sample-efficiency sweep → ablations → figures → export → leave-one-venue-out cross-venue tests.
notebooks/takeoff_validation.ipynb is a focused harness for the takeoff detector: it re-runs the fixed detector over every clip, guards against the old clip-end clustering bug, overlays detected vs hand-marked takeoff frames, and dumps tracks.json / scan caches so the detector and jump segmenter can be tuned on a laptop (no GPU) via src.preprocess.takeoff_local and src.data.jumps_local.
Recommended runtime: A100 (~10 min for SSL training). L4 also fits batch 16. T4 (16 GB) works but drop --batch-size to 4.
Every module is runnable with python -m. To use them without the notebook:
pip install -r requirements.txt # do NOT add torch -- Colab/CUDA wheels would be replaced
# 1. Data: scrape, takeoff-anchored segmentation, track caching, jump ranking.
python -m src.data.scrape --out data/raw
python -m src.data.segment_jumps --raw data/raw --out data/clips --device cuda # takeoff-anchored cuts
python -m src.preprocess.cache_tracks --clips data/clips --out data/annotations/tracks.json --device cuda
python -m src.preprocess.jump_filter --tracks data/annotations/tracks.json \
--out data/annotations/jump_candidates.csv --min-lift 0.02
# (src.data.segment does fixed-length 2s cuts; src.data.filter_clips is the standalone presence filter)
# 2. Annotation (Colab widgets in colab_final.ipynb produce outcomes.csv and fences.csv).
python -m src.preprocess.annotate --clips data/clips --out data/annotations/fences.csv # OpenCV, needs a display
# 3. Automated fence detection (YOLO box + vertical/oxer CNN).
python -m src.preprocess.fence_yolo export --clips data/clips --fences data/annotations/fences.csv \
--out data/fence_yolo
python -m src.preprocess.fence_yolo train --data data/fence_yolo/dataset.yaml --epochs 100 --device 0
python -m src.preprocess.fence_type_cnn train --clips data/clips \
--fences data/annotations/fences.csv --out checkpoints/fence_type.pt
# 4. YOLO horse + geometric d -> auto.csv, then merge into the label table.
python -m src.preprocess.run_pipeline --clips data/clips --fences data/annotations/fences.csv \
--out data/annotations/auto.csv --device cuda
python -m src.data.labels --auto data/annotations/auto.csv --outcomes data/annotations/outcomes.csv \
--by-video data/annotations/by_video.csv --out data/annotations/labels.csv
# 5. SSL pretraining with the venue-confound fixes (balanced sampler + DANN head), then embeddings.
python -m src.ssl.train --clips data/clips --out checkpoints --kinetics-init \
--balance-venues --lambda-domain 0.3
python -m src.ssl.embed --clips data/clips --ckpt checkpoints/encoder.pt --out data/embeddings.npz
# 6. Downstream heads + the three baselines (outcome classifier + d regressor).
python -m src.downstream.train --labels data/annotations/labels.csv --clips data/clips \
--ckpt checkpoints/encoder.pt --finetune --task both --group-by-venue
python -m src.baselines.from_scratch --labels data/annotations/labels.csv --clips data/clips --group-by-venue
python -m src.baselines.imagenet2d --labels data/annotations/labels.csv --clips data/clips --group-by-venue
python -m src.baselines.type_logreg --labels data/annotations/labels.csv --group-by-venue
# 7. Analysis: sample-efficiency sweep, ablations, leave-one-venue-out, then all figures.
python -m src.eval.sample_efficiency --labels data/annotations/labels.csv --clips data/clips \
--ckpt checkpoints/encoder.pt --out data/results/sample_efficiency.csv --group-by-venue
python -m src.eval.ablations --labels data/annotations/labels.csv --clips data/clips \
--type-ckpt checkpoints/encoder.pt --ckpts both=checkpoints/encoder.pt --group-by-venue \
--out data/results/ablations.csv
python -m src.eval.cross_venue --labels data/annotations/labels.csv --clips data/clips \
--ckpt checkpoints/encoder.pt --test A --finetune \
--csv-out data/results/cross_venue_A.csv --fig-out milestone/figures/cross_venue_A.png
python -m src.viz.make_figures --csv data/annotations/auto.csv --labels data/annotations/labels.csv \
--emb data/embeddings.npz --log checkpoints/train_log.csv \
--results data/results/sample_efficiency.csv --out milestone/figures--group-by-venue holds out entire source videos for validation — the honest cross-venue
split the project calls for. Drop it for the easier stratified split (which leaks venue cues
and inflates scores; src.eval.cross_venue --test A measures exactly that gap).