Washington Post source + catalog/store/archive lifecycle model

[kelchm]

What

Adds The Washington Post as a source (Broadsheet's first non-freedomforum provider), cleans up dead freedomforum papers, and introduces the catalog↔store↔archive lifecycle model so catalog changes reach existing installs and a dropped paper's history stays intact. Also records an adversarial architecture boundary review (diagnosis only).

The branch is a deliberately-ordered stack — the bottom commits are solid and independent; the lifecycle-model commit is kept isolated because the review flags that area for a future refactor (see below).

Commits

Commit	Tier
chore: ignore the .claude tooling directory	housekeeping
docs: record the architecture boundary review	solid
fix(freedomforum): count an unexpected upstream status as a failed probe	solid
feat(provider): add Washington Post source	solid
chore(catalog): repoint dc-wp to washingtonpost; drop 81 dead papers	solid
feat: catalog↔store↔archive lifecycle model	on review's RC1/RC2 turf
fix(review): address Copilot + CodeRabbit review findings	review follow-up

Highlights

Washington Post provider. WaPo publishes its print front page as per-page PDFs on an open CloudFront CDN, keyed by full date with a zone code (SU/RE) that rotates day to day. The provider probes both zones across a UTC day window, treats 403 (S3 AccessDenied) as absent, and takes the edition date from the URL folder. The "today's paper" HTML page is Akamai bot-protected, so the provider talks only to the CDN. dc-wp is repointed from the broken freedomforum entry (DC_WP 404s on freedomforum's CDN).

Dead-paper cleanup. Audited all freedomforum papers against two signals — freedomforum's live sitemap inventory and the PDF CDN — and removed the 80 gone from both, plus one malformed entry (bra^pe-jdc). Kept the 11 that still have an active image feed but a dropped PDF. Catalog 606 → 525.

Lifecycle model. Two intertwined changes that meet at loadSources. First, reconcile the store to the catalog on every boot (store.SeedSources): upsert catalog-owned wiring while preserving the user's enabled toggle, and prune papers the catalog dropped — so a catalog fix (repoint, rename, removal) reaches existing installs, not just fresh ones. Second, make the archive independent and self-describing: a paper dropped from the catalog keeps its history browsable and renderable (knownSource is archive-aware) until it ages out on retention, and each source dir carries a .meta.json sidecar (stamped by the reconciler, read by the archive browser) so the archive is portable — drop a <id>/ dir into any install and it shows up named. Labels are stamped before the prune so a paper dropped this release keeps its name; if a label write fails, the prune is skipped that boot so no name is lost.

Architecture review

docs/architecture-review.md records an adversarial audit of the provider/catalog/database/archive seams: four root causes (RC1, the split-ownership sources row, is central), what to preserve, a tiered course-correction, and a forward plan for user-defined sources. Diagnosis only — no structural changes here. The lifecycle commit is correct and preserve-listed, but the eventual source_state refactor will restructure that area; it's isolated so that rework rebases cleanly.

Testing

go build ./..., go vet ./..., go test ./..., and golangci-lint run ./... are all clean at the tip. Every commit compiles (bisectable history). The WaPo provider was verified live end-to-end (fetched and rendered a real front page); the store-reconcile and archive-portability paths are covered by new unit tests plus an end-to-end upgrade-migration check. Copilot and CodeRabbit review findings have been addressed.

🤖 Generated with Claude Code

https://claude.ai/code/session_01Wd6XKSHR9dJpr8hAZjAkcM

[coderabbitai]

📝 Walkthrough

Walkthrough

The PR adds a Washington Post CDN provider, catalog-to-store reconciliation, archive display-name sidecars, archived-source addressability, and related tests and documentation. It also updates archive UI name fallback behavior and ignores Claude Code state.

Changes

Source acquisition and archive flow

Layer / File(s)	Summary
Washington Post provider integration internal/provider/washingtonpost/*, internal/registry/*, internal/catalog/catalog.json, internal/provider/freedomforum/freedomforum.go	Adds conditional CDN polling, PDF validation, provider registration, catalog wiring, and failure handling with comprehensive tests.
Catalog-to-store reconciliation internal/store/*	Refreshes catalog-owned source fields, preserves user enablement, and prunes removed sources with dependent state.
Archive metadata and cleanup internal/archive/*, internal/reconcile/*	Persists source display names in .meta.json, removes empty source directories during pruning, and labels archives after successful reconciliation.
Engine archive integration and addressability pkg/broadsheet/*, cmd/broadsheet-server/ui.go	Labels archives at startup, exposes archive names, preserves access to archived sources, and adds archive-name UI fallback behavior.
Architecture documentation and repository support docs/*, .gitignore	Documents reconciliation, archive portability, provider behavior, architecture findings, and Claude Code state exclusions.

Estimated code review effort: 4 (Complex) | ~60 minutes

Sequence Diagram(s)

sequenceDiagram
  participant Engine
  participant Reconciler
  participant WashingtonPost
  participant CloudFrontCDN
  participant Archive
  Engine->>Reconciler: Reconcile configured source
  Reconciler->>WashingtonPost: Poll candidate dates and zones
  WashingtonPost->>CloudFrontCDN: Send conditional PDF requests
  CloudFrontCDN-->>WashingtonPost: Return edition or probe status
  WashingtonPost-->>Reconciler: Return editions and ETags
  Reconciler->>Archive: Store editions and display name
  Engine-->>Archive: Resolve archived source name

Loading

🚥 Pre-merge checks | ✅ 4 | ❌ 1

❌ Failed checks (1 warning)

Check name	Status	Explanation	Resolution
Docstring Coverage	⚠️ Warning	Docstring coverage is 29.63% which is insufficient. The required threshold is 80.00%.	Write docstrings for the functions missing them to satisfy the coverage threshold.

✅ Passed checks (4 passed)

Check name	Status	Explanation
Linked Issues check	✅ Passed	Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check	✅ Passed	Check skipped because no linked issues were found for this pull request.
Title check	✅ Passed	The title clearly names the new Washington Post source and the lifecycle model change, which matches the main changeset.
Description check	✅ Passed	The description is directly about the Washington Post provider, catalog cleanup, and catalog-store-archive lifecycle changes in this PR.

✨ Finishing Touches

📝 Generate docstrings

• Create stacked PR
• Commit on current branch

---

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

• X
• Mastodon
• Reddit
• LinkedIn

_{Comment @coderabbitai help to get the list of available commands.}

[Copilot]

Pull request overview

This PR adds The Washington Post as a new provider/source, removes a large set of dead Freedom Forum catalog entries, and introduces a catalog↔store↔archive lifecycle model so catalog updates reconcile into existing installs while archived history remains browsable even after a paper is dropped from the catalog.

Changes:

• Add a washingtonpost provider with conditional polling against WaPo’s CloudFront CDN and registry/catalog wiring.
• Reconcile the SQLite sources table to the embedded catalog on every boot (upsert wiring + prune dropped papers while preserving user enabled).
• Make the archive self-describing via per-source .meta.json labels and expose archive names to the UI for dropped papers.

Reviewed changes

Copilot reviewed 18 out of 19 changed files in this pull request and generated 2 comments.

Show a summary per file

File	Description
pkg/broadsheet/rotation_test.go	Updates test fixture helper usage after uniformPNG signature change.
pkg/broadsheet/broadsheet.go	Builds archive earlier, stamps archive labels on startup, expands addressability to archived-only ids, and exposes ArchiveName.
pkg/broadsheet/broadsheet_test.go	Adds tests for archived history outliving catalog membership and startup label backfill/ordering.
internal/store/store.go	Changes SeedSources to a full reconcile (upsert wiring + prune dropped sources + dependent state).
internal/store/store_test.go	Adds coverage for reconcile semantics (preserve user enabled, prune dropped papers, empty seed no-op).
internal/registry/registry.go	Registers and decodes the new washingtonpost provider type.
internal/registry/registry_test.go	Adds decode tests for Washington Post configs (nil/empty/override/malformed).
internal/reconcile/reconcile.go	Stamps archive label on successful archive write.
internal/reconcile/reconcile_test.go	Ensures archive label stamping is exercised in reconcile tests.
internal/provider/washingtonpost/washingtonpost.go	Implements WaPo provider polling and PDF sniffing with conditional GET + multi-zone probing.
internal/provider/washingtonpost/washingtonpost_test.go	Adds thorough unit tests for WaPo polling behavior (cold/warm/403/5xx/non-PDF).
internal/provider/freedomforum/freedomforum.go	Treats unexpected upstream HTTP statuses as failed probes (counts toward failure gating).
internal/catalog/catalog.json	Repoints dc-wp to washingtonpost and removes dead/malformed Freedom Forum entries.
internal/archive/archive.go	Adds .meta.json metadata sidecar with SetName/Name and reclaims emptied source dirs during prune.
internal/archive/archive_test.go	Tests archive label behavior and source-dir reclamation on prune.
docs/architecture.md	Documents reconcile semantics and archive self-description/portability; describes WaPo provider.
docs/architecture-review.md	Adds the recorded adversarial architecture boundary review (diagnosis only).
cmd/broadsheet-server/ui.go	Uses ArchiveName fallback to label dropped papers in the archive UI.
.gitignore	Ignores .claude/ tooling directory.

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

[coderabbitai]

Actionable comments posted: 2

🧹 Nitpick comments (1)

internal/store/store_test.go (1)

103-150: 🗄️ Data Integrity & Integration | 🔵 Trivial | ⚡ Quick win

Exercise crop_overrides pruning too.

The test claims dependent state is removed, but only verifies versions and fetch events. Add a crop override for c and assert it is deleted so every new cleanup statement is covered.

🤖 Prompt for AI Agents

Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@internal/store/store_test.go` around lines 103 - 150, Extend
TestSeedSources_PrunesDroppedPapers by creating a crop override for source "c"
before reseeding the reduced catalog, then query the relevant crop-override
state afterward and assert no override remains for "c". Keep the existing
versions and health-history assertions unchanged.

🤖 Prompt for all review comments with AI agents

Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Inline comments:
In `@docs/architecture.md`:
- Around line 182-200: Update the Washington Post provider documentation to
state that missing CDN objects typically return 403, but CloudFront may also
return 404. Replace the absolute “not a 404” wording in the “Adding a provider”
section while preserving the surrounding explanation of provider-specific
handling.

In `@pkg/broadsheet/broadsheet.go`:
- Line 319: Update stampArchiveLabels to propagate failures from SetName and
ListSources instead of discarding them, and handle that error in SeedSources by
aborting startup before source reconciliation or deletion runs. Preserve normal
archive-label backfilling and pruning only after all label operations complete
successfully.

---

Nitpick comments:
In `@internal/store/store_test.go`:
- Around line 103-150: Extend TestSeedSources_PrunesDroppedPapers by creating a
crop override for source "c" before reseeding the reduced catalog, then query
the relevant crop-override state afterward and assert no override remains for
"c". Keep the existing versions and health-history assertions unchanged.

🪄 Autofix (Beta)

Fix all unresolved CodeRabbit comments on this PR:

• Push a commit to this branch (recommended)
• Create a new PR with the fixes

---

ℹ️ Review info

⚙️ Run configuration

Configuration used: Organization UI

Review profile: CHILL

Plan: Pro

Run ID: 3eda96c8-a322-42bc-9ab2-5235c907c406

📥 Commits

Reviewing files that changed from the base of the PR and between 83262ea and b789781.

📒 Files selected for processing (19)

• .gitignore
• cmd/broadsheet-server/ui.go
• docs/architecture-review.md
• docs/architecture.md
• internal/archive/archive.go
• internal/archive/archive_test.go
• internal/catalog/catalog.json
• internal/provider/freedomforum/freedomforum.go
• internal/provider/washingtonpost/washingtonpost.go
• internal/provider/washingtonpost/washingtonpost_test.go
• internal/reconcile/reconcile.go
• internal/reconcile/reconcile_test.go
• internal/registry/registry.go
• internal/registry/registry_test.go
• internal/store/store.go
• internal/store/store_test.go
• pkg/broadsheet/broadsheet.go
• pkg/broadsheet/broadsheet_test.go
• pkg/broadsheet/rotation_test.go