Repository navigation
observations: immutable, content-addressed, replayable - #22
Merged
Merged
Conversation
Adjudication is the one stochastic tier in this system. This adds the harness that makes it reviewable: hold the inputs fixed, change the prompt version, and diff the outputs. observations Keyed by a SHA-256 hash of the normalized adapter output. Nothing per-fetch is in the hashed payload -- observed_at is a column precisely because it is not part of what the content is -- so re-fetching an unchanged document is a no-op, not a second row. Immutability is enforced twice: an ORM before_update guard raising ObservationIsImmutable, and a BEFORE UPDATE trigger that refuses an UPDATE issued from the sqlite3 shell or a hand-written migration. A guarantee that only holds for code going through the ORM is a convention, not a guarantee. One write path: src/sources/observation.py::store_observation, called from the adapter store path every source in src/sources/ already goes through. Asserted against the source tree, not against intent. evidence Links (observation, watch) with a direction and a magnitude, and records the prompt version on every row. Per row, not per run: a version recorded once in a log cannot answer "where did this judgement come from" six weeks later. insightweaver replay --prompt-version X Rebuilds evidence from observations alone and prints the diff. --commit is required to persist. A replay that wrote by default would destroy the before-state at exactly the moment you wanted to compare against it. Committing is insert-and-delete scoped to one version, and refuses outright if a version disagrees with its own stored rows -- that is nondeterminism, and overwriting it would erase the evidence of the problem the harness exists to surface. Needs no ANTHROPIC_API_KEY. The adjudication prompt is task 016; the only version shipped is null-v0, which returns no verdicts. An adjudicator that guessed a direction from a keyword match would put fabricated judgements into the table built to make judgements reviewable. near-duplicate detection MinHash over 5-word shingles, signature stored beside each observation so a grouping computed now and one computed after a replay agree by construction. Threshold is config (default 0.7); the signature width and shingle size are format, not preference, and are constants with that warning attached. Tested on four real articles pulled from the corpus: a press release carried by two feeds differing only in byline and footer scores 0.89, and the hardest distinct pair -- same publisher, same subject, same boilerplate -- scores 0.016. articles versus observations They coexist under one stated rule, written in src/database/models.py: articles is the pre-rewrite archive and keeps the row shape the older code reads; observations is authoritative for everything built from here on. Each new article gets an observation in the same transaction, linked by observations.article_id. The 55,249 existing rows are not migrated -- the hash is a pure function of columns they already carry, so a backfill is mechanical and belongs to its own task. The migration asserts articles is untouched. Migration is additive: make db-add-observations.
Task 014's reviewer confirmed the one-write-path invariant holds as shipped and flagged the residual: RSSFetcher.fetch_and_store_feed writes articles directly with no observation, and fetch_all_active_feeds still reaches it. It has no in-repo caller since task 012 deleted the orchestrator, which is precisely what makes it dangerous -- it is live, exported, and the obvious thing to reach for when refreshing the corpus. I ran it twice on 2026-08-28. Queued as 025 and marked BLOCKS 015, because routing reads observations: a corpus refreshed through the legacy path produces articles that can never route, and the failure is silent -- the coverage gap would look like a source problem rather than an ingestion one.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Implements
backlog/014. 652 tests (554 → 652),make checkexit 0.Why this one matters more than it looks
The system has exactly one stochastic component — Tier 2 adjudication. Every other tier is deterministic and unit-testable. This is the only mechanism that makes the stochastic one reviewable: Observations immutable and content-addressed, Evidence a pure function of (Observations, prompt version), so a prompt change can be replayed across the corpus and diffed.
It would be worthless if replay were not reproducible, and it would fail in the direction of looking fine. So all three properties were verified independently rather than read.
Immutability refuses at the database, not the ORM
A
BEFORE UPDATEtrigger, so it holds against raw SQL. Re-storing identical content is a no-op returningcreated=False, not an update.Replay is deterministic, and
--commitis requiredcommit()additionally raisesNondeterministicReplayand writes nothing if a version disagrees with its own stored rows.The
articlesrule, written where both tables liveThe task demanded a decision, not an assumption. Coexistence, with one direction of authority:
articles(55,249 rows) is a frozen pre-rewrite archive. Not deleted, not migrated, not read by anything built from 014 onward.observationsis authoritative going forward.Observation, and a test greps the source tree to assert it — the reviewer verified that test would catch a second write path.Chosen over superseding (which means migrating 55k rows in a task whose scope forbids it) and over wrapping (two shapes for one concept, with no answer to "which do I query").
MinHash, tested against real corpus items
Threshold 0.7, 128 permutations, shingle size 5. All four items pulled from the corpus, not written for the test — the reviewer resolved the article IDs and confirmed the titles and feeds match:
0.7 sits in an 0.87-wide gap. Signature width and shingle size are module constants, not config: changing either silently invalidates every stored signature.
A deliberate refusal worth reading
The IC did not ship a keyword-matching baseline adjudicator, and argued against adding one:
The only registered version is
null-v0, which returns nothing. Replay of an unregistered version fails loudly and names what is available. The harness is fully exercised end-to-end by test stubs. Task 016 is where a real adjudicator belongs.Residual, now queued as
backlog/025RSSFetcher.fetch_and_store_feedwrites articles directly with no observation, andfetch_all_active_feedsstill reaches it. It has had no in-repo caller since task 012 deleted the orchestrator — which is exactly what makes it dangerous: it is live, exported, and the obvious thing to reach for when refreshing the corpus. I ran that path twice on 2026-08-28.Queued as 025 and marked BLOCKS 015, because routing reads observations: a corpus refreshed through the legacy path yields articles that can never route, and the failure is silent — the coverage gap would look like a source problem rather than an ingestion one.
The IC disclosed this rather than widening its own diff, which was right.