Transcript pipeline improvements: SST context, live captions, editorial rules

[mriechers]

Summary

• SST context expansion: Pipeline now fetches Social Media Description and follows Project linked records for series-level cast info. Auto-parses speaker names from SST text fields when Host/Presenter aren't set explicitly.
• Live caption handling: Analyst, formatter, and manager agents all detect live captioning input (>> markers, garbled text) and avoid fabricating names from phonetic fragments. Verified against SST data instead.
• PBS Wisconsin house style: Formatter enforces Capitol/capital, OK, lowercase liberals/conservatives, Legislature caps, no oxford commas, abbreviated honorifics, sentence-boundary filler removal, content preservation (no suppression of mild language).
• Chunked formatter fix: Strips Model/Creator/Agent attribution lines that were appearing mid-transcript after chunk merge.
• Timestamp improvements: Lowered auto-trigger from 30min to 10min. Added chaptering best practices from /timestamps skill — duration-based caps, "Episode intro" first chapter, sentence case, topic-not-format naming.
• Wisconsin reference: New knowledge/wisconsin_reference.md with commonly misspelled place names, political figures, legal cases, and program hosts.
• Airtable API key: Configured in Docker .env (not committed) so all jobs now get SST context.

Context

Job 44 (6POL0102, Inside Wisconsin Politics) exposed that the pipeline fabricated "Guy Wagtendonk" as the host name from garbled live caption text. The actual host is Shawn Johnson. Root cause: no Airtable SST context was available (API key missing from Docker), and no agent was trained to handle live captioning artifacts. Comparison with the human editor's final versions confirmed additional gaps in content preservation and speaker attribution accuracy.

Test plan

• Core tests pass (334 passed, queue router rate-limit flakes are pre-existing)
• Job 45 completed successfully with SST context populated (airtable_record_id, media_id, duration all set)
• Timestamp phase triggered for 17-min content (was skipped before at 30-min threshold)
• Speaker pre-parsing extracts "Host: Shawn Johnson" and "Presenter: Zac Schultz, Anya van Wagtendonk, Rich Kremer" from IWP SST data
• Full pipeline re-run after rebuild to verify formatting rules and speaker attribution
• Human editorial review of next IWP episode against editor baseline

🤖 Generated with Claude Code

[mriechers]

Code review

Found 1 issue:

1. knowledge/ directory not copied in Dockerfiles -- the PR adds KNOWLEDGE_DIR = Path("knowledge") and loads knowledge/wisconsin_reference.md at runtime for analyst and formatter phases. Neither Dockerfile.api nor Dockerfile.worker includes a COPY knowledge/ knowledge/ directive. The code uses a safe if wi_ref_path.exists(): guard so it won't crash, but the Wisconsin proper-noun reference will silently never load in Docker deployments. Since the pipeline runs in Docker, this feature is effectively dead in production.

cardigan/api/services/worker.py

Lines 2228 to 2232 in 0f017ef

	if phase_name in ("analyst", "formatter"):
	wi_ref_path = KNOWLEDGE_DIR / "wisconsin_reference.md"
	if wi_ref_path.exists():
	wi_reference = f"\n## Wisconsin Proper-Noun Reference\n\n{wi_ref_path.read_text()}\n\n"

Relevant Dockerfiles to update:

• Dockerfile.api
• Dockerfile.worker

🤖 Generated with Claude Code

_{- If this code review was useful, please react with 👍. Otherwise, react with 👎.}