Skip to content

observations: immutable, content-addressed, replayable - #22

Merged
sam-aydlette merged 2 commits into
mainfrom
task/014-observations
Aug 31, 2026
Merged

sam-aydlette merged 2 commits into
mainfrom
task/014-observations

Conversation

@sam-aydlette

Copy link
Copy Markdown
Owner

Implements backlog/014. 652 tests (554 → 652), make check exit 0.

Why this one matters more than it looks

The system has exactly one stochastic component — Tier 2 adjudication. Every other tier is deterministic and unit-testable. This is the only mechanism that makes the stochastic one reviewable: Observations immutable and content-addressed, Evidence a pure function of (Observations, prompt version), so a prompt change can be replayed across the corpus and diffed.

It would be worthless if replay were not reproducible, and it would fail in the direction of looking fine. So all three properties were verified independently rather than read.

Immutability refuses at the database, not the ORM

UPDATE observations SET payload='{"tampered":true}' WHERE content_hash=…
  -> Error: observations are immutable: the content hash is the identity…

UPDATE observations SET source_id=999;        (no WHERE, every row)
  -> same error

rows after: 4  (unchanged)

A BEFORE UPDATE trigger, so it holds against raw SQL. Re-storing identical content is a no-op returning created=False, not an update.

Replay is deterministic, and --commit is required

same prompt version, two runs -> byte-identical (md5 match)

replay --prompt-version v1            -> "Nothing was written. Re-run with --commit…"
evidence rows: 3 -> 3                    (three verdicts computed, zero written)

replay --prompt-version v2 --against v1
  ~ sha256:31afaa7c… / clerk-wedding-events-expand:
      was [supports 1.000  matched 2 of 2 term(s)]
      now [contradicts 0.500  matched 2 of 2 term(s)]

commit() additionally raises NondeterministicReplay and writes nothing if a version disagrees with its own stored rows.

The articles rule, written where both tables live

The task demanded a decision, not an assumption. Coexistence, with one direction of authority:

  • articles (55,249 rows) is a frozen pre-rewrite archive. Not deleted, not migrated, not read by anything built from 014 onward.
  • observations is authoritative going forward.
  • Exactly one code path constructs an Observation, and a test greps the source tree to assert it — the reviewer verified that test would catch a second write path.
  • The legacy rows have no observation. Stated as a bounded gap, not an ambiguity: the hash is a pure function of columns those rows already carry, so the backfill is mechanical and belongs to its own task.

Chosen over superseding (which means migrating 55k rows in a task whose scope forbids it) and over wrapping (two shapes for one concept, with no answer to "which do I query").

MinHash, tested against real corpus items

Threshold 0.7, 128 permutations, shingle size 5. All four items pulled from the corpus, not written for the test — the reviewer resolved the article IDs and confirmed the titles and feeds match:

pair similarity
same press release, two publishers (differ by a photo credit and an "appeared first on" footer) 0.891
hardest distinct pair — same publisher, same subject, same boilerplate, names the same event 0.016
plainly distinct 0.000

0.7 sits in an 0.87-wide gap. Signature width and shingle size are module constants, not config: changing either silently invalidates every stored signature.

A deliberate refusal worth reading

The IC did not ship a keyword-matching baseline adjudicator, and argued against adding one:

Making up a direction would write fabricated judgements into the table whose purpose is making judgements reviewable.

The only registered version is null-v0, which returns nothing. Replay of an unregistered version fails loudly and names what is available. The harness is fully exercised end-to-end by test stubs. Task 016 is where a real adjudicator belongs.

Residual, now queued as backlog/025

RSSFetcher.fetch_and_store_feed writes articles directly with no observation, and fetch_all_active_feeds still reaches it. It has had no in-repo caller since task 012 deleted the orchestrator — which is exactly what makes it dangerous: it is live, exported, and the obvious thing to reach for when refreshing the corpus. I ran that path twice on 2026-08-28.

Queued as 025 and marked BLOCKS 015, because routing reads observations: a corpus refreshed through the legacy path yields articles that can never route, and the failure is silent — the coverage gap would look like a source problem rather than an ingestion one.

The IC disclosed this rather than widening its own diff, which was right.

Adjudication is the one stochastic tier in this system. This adds the
harness that makes it reviewable: hold the inputs fixed, change the
prompt version, and diff the outputs.

observations
  Keyed by a SHA-256 hash of the normalized adapter output. Nothing
  per-fetch is in the hashed payload -- observed_at is a column
  precisely because it is not part of what the content is -- so
  re-fetching an unchanged document is a no-op, not a second row.
  Immutability is enforced twice: an ORM before_update guard raising
  ObservationIsImmutable, and a BEFORE UPDATE trigger that refuses an
  UPDATE issued from the sqlite3 shell or a hand-written migration. A
  guarantee that only holds for code going through the ORM is a
  convention, not a guarantee.

  One write path: src/sources/observation.py::store_observation, called
  from the adapter store path every source in src/sources/ already goes
  through. Asserted against the source tree, not against intent.

evidence
  Links (observation, watch) with a direction and a magnitude, and
  records the prompt version on every row. Per row, not per run: a
  version recorded once in a log cannot answer "where did this
  judgement come from" six weeks later.

insightweaver replay --prompt-version X
  Rebuilds evidence from observations alone and prints the diff.
  --commit is required to persist. A replay that wrote by default would
  destroy the before-state at exactly the moment you wanted to compare
  against it. Committing is insert-and-delete scoped to one version, and
  refuses outright if a version disagrees with its own stored rows --
  that is nondeterminism, and overwriting it would erase the evidence of
  the problem the harness exists to surface.

  Needs no ANTHROPIC_API_KEY. The adjudication prompt is task 016; the
  only version shipped is null-v0, which returns no verdicts. An
  adjudicator that guessed a direction from a keyword match would put
  fabricated judgements into the table built to make judgements
  reviewable.

near-duplicate detection
  MinHash over 5-word shingles, signature stored beside each observation
  so a grouping computed now and one computed after a replay agree by
  construction. Threshold is config (default 0.7); the signature width
  and shingle size are format, not preference, and are constants with
  that warning attached. Tested on four real articles pulled from the
  corpus: a press release carried by two feeds differing only in byline
  and footer scores 0.89, and the hardest distinct pair -- same
  publisher, same subject, same boilerplate -- scores 0.016.

articles versus observations
  They coexist under one stated rule, written in src/database/models.py:
  articles is the pre-rewrite archive and keeps the row shape the older
  code reads; observations is authoritative for everything built from
  here on. Each new article gets an observation in the same transaction,
  linked by observations.article_id. The 55,249 existing rows are not
  migrated -- the hash is a pure function of columns they already carry,
  so a backfill is mechanical and belongs to its own task. The migration
  asserts articles is untouched.

Migration is additive: make db-add-observations.
Task 014's reviewer confirmed the one-write-path invariant holds as shipped
and flagged the residual: RSSFetcher.fetch_and_store_feed writes articles
directly with no observation, and fetch_all_active_feeds still reaches it.

It has no in-repo caller since task 012 deleted the orchestrator, which is
precisely what makes it dangerous -- it is live, exported, and the obvious
thing to reach for when refreshing the corpus. I ran it twice on 2026-08-28.

Queued as 025 and marked BLOCKS 015, because routing reads observations: a
corpus refreshed through the legacy path produces articles that can never
route, and the failure is silent -- the coverage gap would look like a source
problem rather than an ingestion one.
@sam-aydlette
sam-aydlette merged commit 45dd833 into main Aug 31, 2026
5 checks passed
@sam-aydlette
sam-aydlette deleted the task/014-observations branch August 31, 2026 18:06
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant