LoRA Fine-Tuning of SLMs for Duration-Specific Podcast Chapterization

End-to-end pipeline that ASR-transcribes podcasts, augments transcripts with pause markers, distills hierarchical ToC labels from ToC-LLM, fine-tunes Qwen3-4B with LoRA in a two-stage duration-specialization scheme, and evaluates with hierarchical WindowDiff / ROUGE-L / SBERT metrics.

Python · PyTorch · CUDA · Qwen3-4B-Instruct · QLoRA/PEFT · TRL SFTTrainer · Whisper ASR (unsloth/whisper-large-v3-turbo) · HuggingFace Datasets · pandas/Parquet · NLTK WindowDiff · SBERT

Dataset	YTSEG (784-episode curated subset; 45-episode held-out test)
Paper	Paper (Google Drive)

---

Problem

Most long-form audio — podcasts, YouTube lectures, interviews — ships without chapters. Listeners scrub manually; platforms and search systems cannot index fine-grained segments; downstream summarization and QA tools operate on noisy, unstructured inputs.

Existing chapterization work often trains a single large model across heterogeneous content, assuming structural patterns generalize across genres and durations. We test a different hypothesis: for podcast chapterization, specialization may be more effective than scale. We categorize episodes by duration (ultra-short 1–10 min, short 10–20 min, standard 20–40 min), fine-tune a 4B-parameter SLM with LoRA on each bucket, and compare against a general full-dataset model — targeting competitive quality at lower compute cost.

This repo is a research prototype, not a production serving stack.

---

What We Built

• ASR + prosody pipeline: preprocess/transcribe-audio.py → preprocess/add-pause-markers.py — one sentence per line with inter-sentence pause duration
• Synthetic label generation: ToC-LLM distillation → data/ToC_generated/ (teacher targets for SFT)
• Duration-stratified datasets: preprocess/group-by-duration.py → data/train/{full,ultra_short,short,standard}.parquet; preprocess/select-test-podcasts.py (15 episodes per duration bucket)
• Two-stage LoRA SFT: train_lora_chapterization.py — QLoRA on Qwen3-4B, chat-format SFT (transcript → hierarchical ToC with bracket indices); --finetune-existing for Stage 2 from Stage 1 checkpoint
• Batch inference: run_lora_chapterization.py
• Evaluation harness: evaluate/align-predicted-target-toc.py (PODTILE-style segment matching) + calculate-windowdiff.py, calculate-rougel.py, calculate-sbert.py with ToC-LLM hierarchical wrapper (L=2: chapters + subsections) + plot-metrics.py

---

System Architecture

flowchart TB
    subgraph preprocess [Preprocessing]
        YTSEG[YTSEG audio subset] --> whisper[transcribe-audio.py]
        whisper --> pause[add-pause-markers.py]
        pause --> tocTeacher[ToC-LLM distillation]
        tocTeacher --> parquet[data/train/*.parquet]
    end

    subgraph train [Two-Stage LoRA SFT]
        parquet --> stage1[train_lora -- full.parquet]
        stage1 --> fullCkpt[outputs/lora-chapterization-full]
        fullCkpt --> stage2[train_lora -- duration parquet --finetune-existing]
        stage2 --> durCkpt[ultra_short / short / standard adapters]
    end

    subgraph eval [Evaluation]
        durCkpt --> infer[run_lora_chapterization.py]
        infer --> align[align-predicted-target-toc.py]
        align --> metrics[WindowDiff / ROUGE-L / SBERT]
        metrics --> plots[plot-metrics.py]
    end

Loading

Two-stage training

1. Stage 1 (full): LoRA SFT on all 784 episodes across duration buckets — teaches general transcript → hierarchical ToC formatting.
2. Stage 2 (duration-specific): Continue from Stage 1 adapter; fine-tune separately on ultra-short (367), short (310), and standard (107) subsets.

ToC output format

The model generates a book-style table of contents with hierarchical numbering, section titles, and sentence line indices in square brackets. Pause markers in the input transcript serve as prosodic boundary cues:

1 Introduction [0]
1.1 Outbreak [0]
1.2 Delta variant [10]
2 Vaccines [67]
2.1 Intranasal vaccines [67]

Each transcript line is numbered from 0; the model is instructed to use pause duration to infer chapter vs subsection boundaries.

---

Engineering Decisions

Decision	Rationale
Qwen3-4B-Instruct + 256k context	Instruction-tuned SLM; fits full transcript without chunking (ToC-LLM approach)
QLoRA (4-bit default)	Trainable on ~16GB GPU; LoRA on attention + MLP projections (train_lora_chapterization.py)
Pause marker augmentation	Prosodic silences correlate with topic boundaries (ToC-LLM, PODTILE lineage)
ToC-LLM distillation	No human hierarchical labels at scale; teacher generates synthetic targets
Duration bucketing	Tests specialization hypothesis with structurally distinct episode lengths
Hierarchical metrics	PODTILE WindowDiff + ROUGE-L/SBERT F1 wrapped per ToC-LLM Eq. 1 (L=2)
Parquet train files	Reproducible splits; group-by-duration.py as single source of truth

---

Results

We evaluated four models (full + three duration-specific) on held-out test sets of 15 episodes per duration bucket (45 total). Training split sizes:

Category	Duration	# Examples
Ultra-short	1–10 min	367
Short	10–20 min	310
Standard	20–40 min	107
Total		784

WindowDiff (segmentation; lower is better)

Duration-specific models did not win on their own buckets — the specialization hypothesis was not supported for boundary placement.

Model	Ultra-short	Short	Standard
full	0.0110	0.0073	0.0058
ultra-short	0.0120	0.0067	0.0070
short	0.0145	0.0073	0.0063
standard	0.0077	0.0076	0.0063

ROUGE-L F1 (title lexical overlap; higher is better)

The full model often wins; the short duration category is hardest across all models.

Model	Ultra-short	Short	Standard
full	0.1579	0.0769	0.1053
ultra-short	0.1500	0.0588	0.0833
short	0.1111	0.0481	0.1000
standard	0.1667	0.0588	0.0694

SBERT F1 (title semantic similarity; higher is better)

More promising for specialization — duration-specific models win their buckets for ultra-short and standard categories.

Model	Ultra-short	Short	Standard
full	0.2588	0.1920	0.2039
ultra-short	0.2960	0.1774	0.1650
short	0.2803	0.1974	0.1817
standard	0.2935	0.2025	0.2140

Best model per category (by metric)

Category	Best ROUGE-L F1	Best SBERT F1
Ultra-short	standard model	ultra-short model
Short	full model	standard model
Standard	full model	standard model

ROUGE-L and SBERT selected different winners on every duration bucket — lexical and semantic title quality are not equivalent. Evaluation metric choice materially changes conclusions. Additionally, ToC-LLM distillation targets often collapsed to a single "Transcript" chapter with flat subsections, limiting hierarchical segmentation signal; see the paper §5.5–§6.

---

Repository Map

preprocess/                    # ASR, pause markers, duration splits, test selection
  transcribe-audio.py
  add-pause-markers.py
  group-by-duration.py
  select-test-podcasts.py
train_lora_chapterization.py   # LoRA SFT (Stage 1 + Stage 2)
run_lora_chapterization.py     # batch ToC inference
evaluate/
  align-predicted-target-toc.py
  calculate-windowdiff.py
  calculate-rougel.py
  calculate-sbert.py
  plot-metrics.py
data/
  transcripts_with_pause/      # training transcripts (committed)
  ToC_generated/               # distilled teacher labels
  transcripts_test_with_pause/
  ToC_test/                    # test-set targets
  train/                       # parquet splits (generated locally)
outputs/                       # LoRA checkpoints (gitignored)

data/ytseg-audio/ and outputs/ are gitignored — produce them locally (see preprocess/README.md for YTSEG audio download and HF cache setup).

---

Running the Pipeline

Requirements: NVIDIA GPU + CUDA (QLoRA defaults target ~16GB VRAM) · Python 3.12+

pip install -r requirements.txt

# Preprocess (requires YTSEG audio download — see preprocess/README.md)
python preprocess/transcribe-audio.py
python preprocess/add-pause-markers.py
python preprocess/group-by-duration.py
python preprocess/select-test-podcasts.py

# Stage 1: full model
python train_lora_chapterization.py --parquet data/train/full.parquet

# Stage 2: duration-specific (example)
python train_lora_chapterization.py \
  --parquet data/train/ultra_short.parquet \
  --model outputs/lora-chapterization-full \
  --output-dir outputs/lora-chapterization-ultra-short \
  --finetune-existing

# Inference + evaluation
python run_lora_chapterization.py --model outputs/lora-chapterization-full
python evaluate/align-predicted-target-toc.py
python evaluate/calculate-windowdiff.py
python evaluate/calculate-rougel.py
python evaluate/calculate-sbert.py
python evaluate/plot-metrics.py

Training scripts support additional flags (--no-4bit, --finetune-existing, --max-length, etc.) — run with --help for details. Episodes longer than ~40 minutes were excluded due to GPU OOM constraints (paper §6).

---

Scope & Limitations

• Trained and evaluated on a curated YTSEG subset (784 train / 45 test), not the full 19k-video benchmark.
• Teacher distillation quality is a bottleneck; human-labeled hierarchical targets would be stronger.
• Duration categories may not capture enough structural variance compared to genre or speaker count (future work in paper §6).
• Research prototype — no serving layer, API, or production search index.

---

Authors

Adam Podoxin · Wilson Liang

• Paper: LoRA Fine-Tuning of SLMs for Duration-Specific Podcast Chapterization (Google Drive)
• GitHub: https://github.com/AdamPodoxin/llm-lora-chapterization

---

Dev Notes

Train a model on full dataset and save the adapter

What train_lora_chapterization.py does

Goal: Fine-tune a Qwen3 Instruct model with LoRA so it learns to turn a video transcript (with pause markers) into a table-of-contents string (toc: chapters, bracket indices).

Steps:

1. Load data — Reads a Parquet file (default: data/train/full.parquet) and checks for transcript and toc.

2. Split — Holds out a fraction of rows (default 10%) for validation; uses a fixed seed for reproducibility.

3. Format for chat SFT — Each row becomes a short conversation:

   • system: fixed instructions (generate ToC in the same style as the labels).
   • user: the transcript.
   • assistant: the gold toc.

4. Model — Loads Qwen/Qwen3-4B-Instruct-2507 (or --model). Base weights can be full bf16 on GPU or 4-bit with --use-4bit (QLoRA).

5. LoRA — Attaches PEFT LoRA adapters on the usual linear layers (attention projections + MLP gate/up/down); only those adapter weights are trained, not the full base model.

6. Training — Uses TRL SFTTrainer with assistant_only_loss (by default) so loss is on the assistant / ToC tokens, not the system+user prompt. Runs for a few epochs, evaluates each epoch, saves the best checkpoint by eval loss.

7. Output — Writes the trained adapter + tokenizer under outputs/lora-chapterization/ (or --output-dir).

CLI — You can change Parquet path, output dir, epochs, LR, batch size, LoRA rank, max sequence length, etc.; --no-assistant-only-loss turns off assistant-only masking if your setup errors on it.

How to finetune from disk instead of base Qwen

Run the command python3 train_lora_chapterization.py --model PATH/TO/LOCAL/PRETRAINED/MODEL --output-dir PATH/TO/OUTPUT/FINETUNED/MODEL --parquet PATH/TO/TRAINING/PARQUET/FILE --finetune-existing

Problem 1

Can't set assistant_only_loss=True this is no ideal.

transcript + toc is exactly what you need. The failure was not “wrong columns”; it was how TRL + Qwen3’s built-in chat template interact when you use the messages format with assistant_only_loss=True: that path expects the template to include a {% generation %} Jinja block so the tokenizer can mark assistant tokens. Qwen3’s default template doesn’t, so masking fails.

What you can do (without changing the Parquet schema):

1. Use prompt + completion instead of messages
   Build each example as:

   • prompt: chat turns up to and including the user (system + user with transcript), in the same role/content structure TRL expects.
   • completion: assistant turn(s) with only the toc text.

   TRL’s prompt–completion path computes a completion_mask by comparing tokenized prompt vs full sequence — it does not rely on {% generation %} in the template. Then you enable completion-only style loss (not the broken assistant-mask path). This is usually the practical fix for models whose template lacks generation.

2. Custom chat template
   Save a copy of Qwen’s chat_template.jinja, add the {% generation %} (or equivalent) around the assistant segment, and point SFTConfig.chat_template_path at it so return_assistant_tokens_mask works. More delicate, but keeps the messages layout.

3. Manual label masking
   Tokenize yourself and set labels to -100 everywhere except assistant tokens. Full control, more code.

So: nothing is wrong with your input file’s content — you can keep transcript / toc and still get supervision only on the ToC by switching dataset layout (prompt/completion) or template, not by changing the Parquet columns.