segue-training-data

Training data and tooling for Segue, an open-weight ad-detection model built for MinusPod. The model learns to find advertisement segments in podcast transcripts, assign each one a category (sponsor, cross_promo, self_promo, interaction), and answer in the JSON format MinusPod's processing pipeline expects.

Two documents cover most questions: docs/training.md to run the training pipeline yourself, and docs/glossary.md for the terms this project uses in training, tooling, and scoring.

Status

Training runs locally now. tools/train_local.py (Transformers + PEFT, BF16 LoRA on a single CUDA GPU) replaced the managed backend, gated by a preflight that renders every example and runs a real optimizer step before it will let a run start. docs/training.md is the end-to-end guide.

Fine-tuning has not beaten the base model. Two runs on Qwen3.5-9B match it on the benchmark corpus: r2 on 8555 machine_accepted windows and ablation-b on 205 human-corrected windows, which land within 0.2s of each other on both boundary metrics. Two datasets 40 times apart in size, one machine and one human, produce the same model, so there is no measured advantage to correction data and no sign that more of it would help. runs/ablation-b/results.md has the numbers.

What did move the score was the prompt. Scoring the corpus in audio seconds showed the model finds 51 of 53 ads and then under-covers them, emitting one span per advertiser instead of one per break. Rewriting that merge rule took F0.5 from 0.754 to 0.798 and recovered 134 seconds of ad audio, in minutes rather than the hours each training run costs. It also cut 133 more seconds of show content, which needs evaluating before the rule ships. It is recorded in runs/prompt-break-merge/results.md and belongs in MinusPod, not here.

Before it, the dataset was corrected against an audit: one span now means one contiguous ad break, spans whose only evidence was audio or whose category the prompt forbids were dropped with provenance, and end_text is validated rather than assumed. MinusPod's benchmark scorer was changed to match, so older scores are not comparable to new ones.

The earlier managed-backend checkpoint (Qwen3.5-4B, tier B, F0.5 0.686) is recorded in runs/20260830-162400/results.md. That number predates the per-break scoring change and should be re-measured before it is compared with anything. Phase 2 widens the dataset; the plan is in docs/phase2-data-plan.md.

What is in here

Path	Contents
data/examples/<feed>/<episode>.jsonl	One training example per transcript window
data/holdout.txt	Benchmark episodes that must never appear in training data
prompts/<sha256>.txt	Deduplicated system prompts referenced by examples
schema/example.schema.json	JSON Schema for a training example
tools/	Extraction, validation, dataset build, training, export, and evaluation scripts
runs/<run-id>/	Config snapshot and scored results for one run
docs/	Training guide, glossary, design, phase 1 runbook, phase 2 data plan
benchmark.local.toml.example	Benchmark harness config for scoring a locally served checkpoint

Each example pairs a fully rendered detection prompt (the same prompt MinusPod sends in production) with the ad markers that survived MinusPod's validation, review, and cutting pipeline for that window. Windows with no ads keep an empty completion on purpose: most of a podcast is not ads, and the model has to learn that too.

Every example carries a provenance tier:

• human_verified: a person reviewed or corrected the markers
• machine_accepted: markers cut in production without human review
• hard_negative: spans a human rejected, kept as counterexamples
• synthetic: generated by augmentation tooling

Tools

All commands run from the repo root with uv. The extractor imports code from a MinusPod checkout; clone MinusPod next to this repo or set SEGUE_MINUSPOD_SRC.

Build a dataset:

# Pull examples out of a MinusPod database copy (never the live file)
uv run python tools/extract.py --db .local/minuspod.db --limit 25

# Check every example: schema, holdout, prompt store, window bounds
uv run python tools/validate.py

# One-shot migration to the per-break output contract (drops outro and
# audio-only-evidence spans, merges sub-15s gaps, fixes end_text)
uv run python tools/fix_labels.py --dry-run

# Build chat-format train/val JSONL; val feeds are held out whole.
# Name two or three feeds, not the whole roster: every feed listed goes to
# val, so passing all of them leaves an empty training file.
uv run python tools/build_dataset.py --val-feeds feed-a,feed-b,feed-c

Train and export on Tinker, the legacy backend (needs TINKER_API_KEY and uv sync --extra train):

uv run python tools/train_tinker.py --list-models
uv run python tools/train_tinker.py --train .local/train.jsonl
uv run python tools/export_model.py --tinker-path "tinker://<run-id>/sampler_weights/final"

Local training

tools/train_local.py is the default training backend: Transformers + PEFT, BF16 LoRA on a single CUDA GPU. Training needs about 48 GB of device memory; the measured peak is 43.1 GiB on the longest window in this dataset.

docs/training.md is the complete guide: hardware, setup, the pinned base revision, every command with its expected output, and a troubleshooting section covering the failures hit during bring-up. Read that rather than assembling the steps from here. Unfamiliar terms are defined in docs/glossary.md. In short:

# Gate: refuses to train until data, renderer, and device check out
uv run python tools/preflight.py --revision $REV

# LoRA-train against the preflight stamp; writes runs/<run-id>/run.json
uv run python tools/train_local.py --run-id r1 --revision $REV

# Score JSON compliance, span P/R/F0.5, and boundary MAE on held-out val
uv run python tools/eval_generation.py --run-id r1 --revision $REV \
    --adapter .local/runs/r1/adapter

# Export the adapter and a merged BF16 model behind an equivalence gate
uv run python tools/export_local.py --run-id r1 --revision $REV \
    --adapter .local/runs/r1/adapter

Serve and check before spending time on a full benchmark run:

# Response shape, finish reason, and latency on real windows
uv run python tools/smoke_test.py --base-url http://<gpu-host>:8123/v1 --ads-only --n 3

# Compare the tinker renderer and chat-template assistant prefixes
uv run python tools/inspect_renderer.py

docs/phase1-runbook.md has the vLLM serving config, the flags a 16 GB card needs, and the failures worth recognizing on the way.

Two layers, scored the same way.

After every training run, tools/eval_generation.py decodes the feed-held-out validation set with the adapter loaded and scores span-level accuracy directly. No serving stack required:

uv run python tools/eval_generation.py --run-id r2 --revision $REV \
  --adapter .local/runs/r2/adapter

It reports JSON compliance, precision, recall, F0.5, false positives on windows that contain no ads, and boundary error, into .local/eval-<run>.json. Held-out loss during training is a progress signal, not this: a model can improve its token loss while getting no better at placing spans.

Before a release, MinusPod's LLM benchmark harness scores the checkpoint against the human-verified corpus, which is what produces a row comparable to the published multi-model table. That path needs the model served over vLLM. Run the harness from a copy, never inside the MinusPod checkout, because it writes to a results directory that is tracked in git and holds the published history:

tools/setup_isolated_benchmark.sh <minuspod> ~/segue-benchmark
cp benchmark.local.toml.example ~/segue-benchmark/benchmarks/llm/benchmark.toml

Set the vLLM base URL in that config, then follow docs/phase1-runbook.md. The corpus episodes are listed in data/holdout.txt, excluded from training by the extractor and checked again by the validator.

Both layers apply the same span policy: predictions and ground truth are merged into contiguous breaks (gaps under 15 seconds) before matching at IoU >= 0.5, and both rank by F0.5 rather than F1, because cutting real content is a worse failure than leaving an ad in. See docs/glossary.md for the terms.

Contributing data

Run the extractor against a copy of your own MinusPod database and open a PR with the resulting data/examples/ files. tools/validate.py must pass. Set --instance to a stable pseudonym and --license to the license of the source shows if you know it.

The categories interaction and cross_promo are the thinnest part of the dataset, so episodes carrying either are the most useful thing to send.