Skip to content

test(sandbox): preserve the disposable ship target as a fixture - #228

Merged
schmug merged 1 commit into
mainfrom
feat/sandbox-ship-fixture
Sep 10, 2026
Merged

schmug merged 1 commit into
mainfrom
feat/sandbox-ship-fixture

Conversation

@schmug

@schmug schmug commented Sep 10, 2026

Copy link
Copy Markdown
Owner

--dry-run rehearses everything except the ship. The four steps it deliberately stops before — the R2 PUTs, the feed-manifest upsert, the Pages deploy hook, the covered.json write — had only ever run against the two live shows. So the one part of the pipeline with irreversible outward effects was the one part with no safe way to exercise it.

This preserves the disposable publishing target that fixes that.

Verified end to end

Ran for real on 2026-09-10 against the real bucket:

r2_status published, public URL HTTP 200
appeared sandbox/<slug>.mp3 (1,896,532 B), sandbox/<slug>.jpg, manifest-sandbox.json
manifest.json sha 74639eed…74639eed…, still 91 entries
manifest-frontier-commits.json sha 26b489c6…26b489c6…, still 2 entries
covered.json 1064 → 1064 entries

Checked against a prediction written before the run, not rationalised after. The deploy hook fired (HTTP 200) and the probe surfaces nowhere on /podcast/, /frontier-commits/, or the real manifest.

Why the test file matters more than the fixture

The fixture is the dangerous kind of file: it points at the real bucket, it gets edited by whoever is spiking, and its entire safety property is four keys holding it away from two live feeds.

tests/test_sandbox_fixture.py re-derives the live namespaces from their own sources — the daily show's constants, and Frontier Commits' documented manifest parsed out of its SKILL.md rather than copied — and fails if the sandbox collides with either. A show that changes its namespace drags these checks with it, instead of silently leaving them protecting nothing.

Every guard is mutation-tested. Each of these fails at least one test:

mutation caught
drop r2_key_prefix 2 failed
slug_prefix → the daily show's 1 failed
feed object → manifest.json 1 failed
key prefix → Frontier Commits' 1 failed
a segment gains a real source_url 1 failed
pin a date 1 failed
ship_modespotify 1 failed
drop the intro role 2 failed
music hash no longer matches 1 failed

Two deliberate choices

  • No date key, so every run mints today's slug. R2 objects are immutable-cached: republishing a slug replaces the origin bytes while the edge keeps serving the old ones per POP, so a pinned date would make every later spike fight a stale cache.
  • Music enabled, so one command exercises the mix and the music pre-flight too. The asset hash is pinned in the test, not only the fixture, so a re-recorded asset fails a unit test rather than a live run's pre-flight mid-spike.

Docs

CLAUDE.md gains a section on when to reach for it — a change touching the ship (feed schema, R2 keys, slug minting, the publish path) — and when not to: audio or script changes, which a dry run covers completely and where a live publish adds only litter. It also records the two things the sandbox does not isolate (the deploy hook fires; the run appends a normal web-ready record to runs.jsonl), and how to reshape it for a future show's spike.

The documented invocation was run verbatim before committing: pre-flight 8/8, artifact gate PASS, previewed URL under the sandbox prefix.

Tests

1549 passing, 0 failing. ruff check + format clean; all 7 pre-commit hooks pass.

🤖 Generated with Claude Code

`--dry-run` rehearses everything except the ship. The four steps it stops
before — the R2 PUTs, the feed-manifest upsert, the Pages deploy hook, and the
covered.json write — had only ever run against the two live shows, so the one
part of the pipeline with irreversible outward effects was the one part with no
safe way to exercise it.

tests/data/sandbox_manifest.json is a disposable publishing target that runs
them for real, in a namespace no feed reads. Verified end to end on 2026-09-10:
r2_status published, public URL HTTP 200, both live feed manifests
byte-identical afterwards (sha 74639eed / 26b489c6) and covered.json unmoved at
1064 entries.

The fixture is the dangerous kind of file — it points at the real bucket, it
gets edited by whoever is spiking, and its whole safety property is four keys
holding it away from two live feeds. tests/test_sandbox_fixture.py is what
keeps that true. It re-derives the live namespaces from their own sources (the
daily show's constants; Frontier Commits' documented manifest, parsed from
SKILL.md rather than copied) and fails if the sandbox collides with either, so
a show that changes its namespace drags these checks with it instead of
silently leaving them protecting nothing.

Every guard is mutation-tested — dropping r2_key_prefix, pointing slug_prefix
at the daily show, aiming the feed object at manifest.json, borrowing Frontier
Commits' key prefix, giving a segment a real source_url, pinning a date,
flipping ship_mode to spotify, dropping the intro role, and breaking the music
hash each fail at least one test.

Two deliberate choices in the fixture:

- No `date` key, so every run mints today's slug. R2 objects are
  immutable-cached: republishing a slug replaces the origin bytes while the
  edge keeps serving the old ones per POP, so a pinned date would make every
  later spike fight a stale cache.
- Music is enabled, so one command also exercises the mix and the music
  pre-flight check. The asset hash is pinned in the test rather than only in
  the fixture, so a re-recorded asset fails a unit test instead of a live run's
  pre-flight, mid-spike.

CLAUDE.md documents when to reach for it (a change touching the ship — feed
schema, R2 keys, slug minting, the publish path) and when not to (audio or
script changes, which a dry run covers completely). The documented invocation
was run verbatim before committing: pre-flight 8/8, artifact gate PASS, and the
previewed URL under the sandbox prefix.

Tests: 1549 passing, 0 failing. ruff check + format clean; all 7 pre-commit
hooks pass.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
@schmug
schmug merged commit e1a352c into main Sep 10, 2026
3 checks passed
@schmug
schmug deleted the feat/sandbox-ship-fixture branch September 10, 2026 21:27
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant