test(sandbox): preserve the disposable ship target as a fixture - #228
Merged
Merged
Conversation
`--dry-run` rehearses everything except the ship. The four steps it stops before — the R2 PUTs, the feed-manifest upsert, the Pages deploy hook, and the covered.json write — had only ever run against the two live shows, so the one part of the pipeline with irreversible outward effects was the one part with no safe way to exercise it. tests/data/sandbox_manifest.json is a disposable publishing target that runs them for real, in a namespace no feed reads. Verified end to end on 2026-09-10: r2_status published, public URL HTTP 200, both live feed manifests byte-identical afterwards (sha 74639eed / 26b489c6) and covered.json unmoved at 1064 entries. The fixture is the dangerous kind of file — it points at the real bucket, it gets edited by whoever is spiking, and its whole safety property is four keys holding it away from two live feeds. tests/test_sandbox_fixture.py is what keeps that true. It re-derives the live namespaces from their own sources (the daily show's constants; Frontier Commits' documented manifest, parsed from SKILL.md rather than copied) and fails if the sandbox collides with either, so a show that changes its namespace drags these checks with it instead of silently leaving them protecting nothing. Every guard is mutation-tested — dropping r2_key_prefix, pointing slug_prefix at the daily show, aiming the feed object at manifest.json, borrowing Frontier Commits' key prefix, giving a segment a real source_url, pinning a date, flipping ship_mode to spotify, dropping the intro role, and breaking the music hash each fail at least one test. Two deliberate choices in the fixture: - No `date` key, so every run mints today's slug. R2 objects are immutable-cached: republishing a slug replaces the origin bytes while the edge keeps serving the old ones per POP, so a pinned date would make every later spike fight a stale cache. - Music is enabled, so one command also exercises the mix and the music pre-flight check. The asset hash is pinned in the test rather than only in the fixture, so a re-recorded asset fails a unit test instead of a live run's pre-flight, mid-spike. CLAUDE.md documents when to reach for it (a change touching the ship — feed schema, R2 keys, slug minting, the publish path) and when not to (audio or script changes, which a dry run covers completely). The documented invocation was run verbatim before committing: pre-flight 8/8, artifact gate PASS, and the previewed URL under the sandbox prefix. Tests: 1549 passing, 0 failing. ruff check + format clean; all 7 pre-commit hooks pass. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
--dry-runrehearses everything except the ship. The four steps it deliberately stops before — the R2 PUTs, the feed-manifest upsert, the Pages deploy hook, thecovered.jsonwrite — had only ever run against the two live shows. So the one part of the pipeline with irreversible outward effects was the one part with no safe way to exercise it.This preserves the disposable publishing target that fixes that.
Verified end to end
Ran for real on 2026-09-10 against the real bucket:
r2_statuspublished, public URL HTTP 200sandbox/<slug>.mp3(1,896,532 B),sandbox/<slug>.jpg,manifest-sandbox.jsonmanifest.json74639eed…→74639eed…, still 91 entriesmanifest-frontier-commits.json26b489c6…→26b489c6…, still 2 entriescovered.jsonChecked against a prediction written before the run, not rationalised after. The deploy hook fired (HTTP 200) and the probe surfaces nowhere on
/podcast/,/frontier-commits/, or the real manifest.Why the test file matters more than the fixture
The fixture is the dangerous kind of file: it points at the real bucket, it gets edited by whoever is spiking, and its entire safety property is four keys holding it away from two live feeds.
tests/test_sandbox_fixture.pyre-derives the live namespaces from their own sources — the daily show's constants, and Frontier Commits' documented manifest parsed out of its SKILL.md rather than copied — and fails if the sandbox collides with either. A show that changes its namespace drags these checks with it, instead of silently leaving them protecting nothing.Every guard is mutation-tested. Each of these fails at least one test:
r2_key_prefixslug_prefix→ the daily show'smanifest.jsonsource_urldateship_mode→spotifyTwo deliberate choices
datekey, so every run mints today's slug. R2 objects are immutable-cached: republishing a slug replaces the origin bytes while the edge keeps serving the old ones per POP, so a pinned date would make every later spike fight a stale cache.Docs
CLAUDE.md gains a section on when to reach for it — a change touching the ship (feed schema, R2 keys, slug minting, the publish path) — and when not to: audio or script changes, which a dry run covers completely and where a live publish adds only litter. It also records the two things the sandbox does not isolate (the deploy hook fires; the run appends a normal
web-readyrecord toruns.jsonl), and how to reshape it for a future show's spike.The documented invocation was run verbatim before committing: pre-flight 8/8, artifact gate PASS, previewed URL under the sandbox prefix.
Tests
1549 passing, 0 failing. ruff check + format clean; all 7 pre-commit hooks pass.
🤖 Generated with Claude Code