From 38d0e9450eb943316b0f9feab5071fedf9044ccc Mon Sep 17 00:00:00 2001 From: sam-aydlette Date: Mon, 31 Aug 2026 12:25:38 -0400 Subject: [PATCH 1/2] observations: immutable, content-addressed, and replayable Adjudication is the one stochastic tier in this system. This adds the harness that makes it reviewable: hold the inputs fixed, change the prompt version, and diff the outputs. observations Keyed by a SHA-256 hash of the normalized adapter output. Nothing per-fetch is in the hashed payload -- observed_at is a column precisely because it is not part of what the content is -- so re-fetching an unchanged document is a no-op, not a second row. Immutability is enforced twice: an ORM before_update guard raising ObservationIsImmutable, and a BEFORE UPDATE trigger that refuses an UPDATE issued from the sqlite3 shell or a hand-written migration. A guarantee that only holds for code going through the ORM is a convention, not a guarantee. One write path: src/sources/observation.py::store_observation, called from the adapter store path every source in src/sources/ already goes through. Asserted against the source tree, not against intent. evidence Links (observation, watch) with a direction and a magnitude, and records the prompt version on every row. Per row, not per run: a version recorded once in a log cannot answer "where did this judgement come from" six weeks later. insightweaver replay --prompt-version X Rebuilds evidence from observations alone and prints the diff. --commit is required to persist. A replay that wrote by default would destroy the before-state at exactly the moment you wanted to compare against it. Committing is insert-and-delete scoped to one version, and refuses outright if a version disagrees with its own stored rows -- that is nondeterminism, and overwriting it would erase the evidence of the problem the harness exists to surface. Needs no ANTHROPIC_API_KEY. The adjudication prompt is task 016; the only version shipped is null-v0, which returns no verdicts. An adjudicator that guessed a direction from a keyword match would put fabricated judgements into the table built to make judgements reviewable. near-duplicate detection MinHash over 5-word shingles, signature stored beside each observation so a grouping computed now and one computed after a replay agree by construction. Threshold is config (default 0.7); the signature width and shingle size are format, not preference, and are constants with that warning attached. Tested on four real articles pulled from the corpus: a press release carried by two feeds differing only in byline and footer scores 0.89, and the hardest distinct pair -- same publisher, same subject, same boilerplate -- scores 0.016. articles versus observations They coexist under one stated rule, written in src/database/models.py: articles is the pre-rewrite archive and keeps the row shape the older code reads; observations is authoritative for everything built from here on. Each new article gets an observation in the same transaction, linked by observations.article_id. The 55,249 existing rows are not migrated -- the hash is a pure function of columns they already carry, so a backfill is mechanical and belongs to its own task. The migration asserts articles is untouched. Migration is additive: make db-add-observations. --- Makefile | 9 +- README.md | 62 ++++ src/cli/app.py | 16 +- src/cli/replay.py | 142 ++++++++ src/config/settings.py | 16 + .../add_observations_and_evidence.py | 110 ++++++ src/database/models.py | 212 ++++++++++- src/evidence/__init__.py | 56 +++ src/evidence/adjudicator.py | 210 +++++++++++ src/evidence/replay.py | 282 +++++++++++++++ src/sources/__init__.py | 23 ++ src/sources/minhash.py | 191 ++++++++++ src/sources/observation.py | 214 ++++++++++++ src/sources/store.py | 40 ++- tests/cli/test_replay_cli.py | 194 +++++++++++ .../test_add_observations_and_evidence.py | 163 +++++++++ tests/database/test_connection.py | 17 +- tests/database/test_models.py | 13 +- tests/evidence/__init__.py | 0 tests/evidence/conftest.py | 27 ++ tests/evidence/stubs.py | 165 +++++++++ tests/evidence/test_replay.py | 290 ++++++++++++++++ tests/sources/fixtures/near_duplicates.json | 38 ++ tests/sources/test_minhash.py | 172 +++++++++ tests/sources/test_observation.py | 328 ++++++++++++++++++ 25 files changed, 2970 insertions(+), 20 deletions(-) create mode 100644 src/cli/replay.py create mode 100644 src/database/migrations/add_observations_and_evidence.py create mode 100644 src/evidence/__init__.py create mode 100644 src/evidence/adjudicator.py create mode 100644 src/evidence/replay.py create mode 100644 src/sources/minhash.py create mode 100644 src/sources/observation.py create mode 100644 tests/cli/test_replay_cli.py create mode 100644 tests/database/test_add_observations_and_evidence.py create mode 100644 tests/evidence/__init__.py create mode 100644 tests/evidence/conftest.py create mode 100644 tests/evidence/stubs.py create mode 100644 tests/evidence/test_replay.py create mode 100644 tests/sources/fixtures/near_duplicates.json create mode 100644 tests/sources/test_minhash.py create mode 100644 tests/sources/test_observation.py diff --git a/Makefile b/Makefile index a42e67d..22a1473 100644 --- a/Makefile +++ b/Makefile @@ -1,4 +1,4 @@ -.PHONY: help install install-dev test lint format fmt typecheck check pre-commit clean clean-all coverage db-add-watches db-drop-briefing update-deps +.PHONY: help install install-dev test lint format fmt typecheck check pre-commit clean clean-all coverage db-add-watches db-add-observations db-drop-briefing update-deps # Tool resolution (added 2026-08-24). # Before this, every recipe called bare `pytest` / `ruff` / `mypy`, which only @@ -45,6 +45,7 @@ help: @echo "" @echo "Database:" @echo " make db-add-watches Create the watches table (additive, safe)" + @echo " make db-add-observations Create observations and evidence (additive, safe)" @echo " make db-drop-briefing Drop the deleted briefing product's tables" @echo " (DESTRUCTIVE; captures to a dump first)" @echo "" @@ -120,6 +121,12 @@ clean-all: clean db-add-watches: $(PYTHON) -m src.database.migrations.add_watches_table +# Additive in the same way, and specifically does NOT touch the 55,249-row +# articles table: observations coexist with it under the rule written in +# src/database/models.py. (2026-08-31, backlog task 014.) +db-add-observations: + $(PYTHON) -m src.database.migrations.add_observations_and_evidence + # Deliberately does NOT pass --confirm. The migration refuses without it, so # running this target prints what it would destroy and stops; the operator types # the real command themselves. A Makefile target that drops tables on `make` is diff --git a/README.md b/README.md index 18e6ebd..345e5fb 100644 --- a/README.md +++ b/README.md @@ -522,6 +522,68 @@ beat can select it. --- +## Observations, Evidence, and replay + +Added 2026-08-31 (`backlog/014`). Adjudication -- deciding whether an item bears +on a pre-registered Watch -- is the only stochastic component in the system. +Everything either side of it is deterministic and unit-testable. This layer is +what makes the stochastic part reviewable: hold the inputs fixed, change the +prompt, and diff the outputs. + +**Observations are immutable and content-addressed.** An observation is one +thing a source published, keyed by a SHA-256 hash of the normalized adapter +output. Nothing per-fetch is in the hashed payload -- not the fetch time, not a +session id -- so re-fetching an unchanged document is a no-op rather than a +second row. Once written, a row cannot be changed: an ORM update raises +`ObservationIsImmutable`, and a `BEFORE UPDATE` trigger refuses an `UPDATE` +issued from anywhere else, including the `sqlite3` shell. + +**Evidence is derived, and every row records the prompt version that produced +it.** `evidence` links `(observation, watch)` with a direction and a magnitude. +The `prompt_version` column is per row, not per run, so "where did this +judgement come from" is answerable six weeks later. + +```bash +# Rebuild v2's evidence and diff it against what v1 stored. Writes nothing. +insightweaver replay --prompt-version v2 --against v1 + +# Re-run v1 against its own stored rows. An empty diff is the reproducibility check. +insightweaver replay --prompt-version v1 + +# Persist. --commit is required; a replay that wrote by default would destroy +# the before-state at the moment you wanted to compare against it. +insightweaver replay --prompt-version v2 --commit +``` + +Replay needs no `ANTHROPIC_API_KEY`. The adjudicator is a plug-in seam +(`src/evidence/adjudicator.py`); the adjudication prompt itself is a later task, +and the only version this build ships is `null-v0`, which returns no verdicts. +An adjudicator that guessed a direction from a keyword match would put +fabricated judgements into the table whose whole purpose is to make judgements +reviewable. + +**Near-duplicate detection.** Two feeds carrying the same wire story with +different boilerplate have different content hashes, correctly. A MinHash +signature over 5-word shingles is written beside each observation and groups +them: `settings.near_duplicate_threshold` (default 0.7, `NEAR_DUPLICATE_THRESHOLD`) +is the tunable. Measured on two real corpus pairs, a genuine near-duplicate +scores 0.89 and the hardest distinct pair 0.016. + +**`observations` versus the 55,249-row `articles` table.** They coexist, with one +stated rule: `articles` is the pre-rewrite archive and keeps the row shape the +older code reads; `observations` is authoritative for everything built from task +014 onward. Exactly one code path writes an observation +(`src/sources/observation.py`, called from the adapter store path), and each new +article gets an observation in the same transaction, linked by +`observations.article_id`. The pre-existing rows have no observation and are not +migrated -- the hash is a pure function of columns they already carry, so a +backfill is mechanical and is left to its own task. + +Create the tables with `make db-add-observations` (additive; it does not touch +`articles`). + +--- + ## Requirements - Python 3.10+ diff --git a/src/cli/app.py b/src/cli/app.py index fc2ee4e..baccb7f 100644 --- a/src/cli/app.py +++ b/src/cli/app.py @@ -19,6 +19,7 @@ from .colors import accent, header, muted from .output import set_debug_mode +from .replay import replay_command from .sources import sources_command from .watch import watch_command @@ -31,12 +32,16 @@ COMMAND_DISPATCH = { "sources": sources_command, "watch": watch_command, + "replay": replay_command, } def print_command_refresher(): """Print a short refresher of available commands.""" - click.echo(f"\n{header('Commands:')} {accent('sources')} | {accent('watch')} | help | exit\n") + click.echo( + f"\n{header('Commands:')} {accent('sources')} | {accent('watch')} | " + f"{accent('replay')} | help | exit\n" + ) def print_help(): @@ -46,6 +51,9 @@ def print_help(): click.echo( f" {accent('watch')} - Pre-registered watches and the decisions they serve" ) + click.echo( + f" {accent('replay')} - Rebuild Evidence from Observations and diff it" + ) click.echo(f" {accent('help')} - Show this help message") click.echo(f" {accent('exit')} - Exit InsightWeaver") click.echo() @@ -64,6 +72,12 @@ def print_help(): muted(" Watches are only ever created from those files -- there is no 'watch add'.") ) click.echo() + click.echo(header("Replay command:")) + click.echo(f" {accent('replay --prompt-version X')} - Rebuild Evidence for prompt version X") + click.echo( + muted(" Prints the diff against stored Evidence and writes nothing. --commit persists.") + ) + click.echo() click.echo(muted("Tip: Add --debug to any command to see detailed logs")) click.echo() diff --git a/src/cli/replay.py b/src/cli/replay.py new file mode 100644 index 0000000..05f640f --- /dev/null +++ b/src/cli/replay.py @@ -0,0 +1,142 @@ +""" +Replay command - rebuild Evidence from Observations and show what changed. + +``--commit`` is required to persist, and that is the whole shape of this +command. A replay that wrote by default would destroy the before-state at +exactly the moment you wanted to compare against it: you would have run the +thing that answers "what did my prompt change do" and, in answering, made the +question unanswerable. Printing the diff and requiring one more flag costs a +keystroke and preserves the property the harness exists to provide. + +Three ways to use it: + + replay --prompt-version v1 + Recompute v1 and diff against v1's stored rows. An empty diff is the + reproducibility check: a deterministic adjudicator changes nothing. + + replay --prompt-version v2 --against v1 + Diff v2's output against v1's stored rows. This is the prompt-change + review. + + replay --prompt-version v2 --commit + Make the stored rows for v2 equal what v2 just produced. Touches no other + version's rows, and refuses if v2 disagrees with itself. + +Added 2026-08-31 for backlog task 014. +""" + +import click + +from ..database.connection import get_db +from ..database.models import Observation, Watch +from ..evidence import ( + NondeterministicReplay, + UnknownPromptVersion, + commit, + diff, + format_diff, + load_adjudicator, + rebuild, + resolve, + stored_evidence, +) +from .colors import accent, header, muted, warning + + +def _adjudicator_for(prompt_version: str, path: str | None): + """ + The adjudicator to run, and a check that it is the one being labelled. + + A ``--adjudicator`` whose own ``prompt_version`` disagrees with + ``--prompt-version`` is refused rather than reconciled. The label written on + every evidence row has to be the name the adjudicator answers to, or the + column is decoration. + """ + try: + adjudicator = load_adjudicator(path) if path else resolve(prompt_version) + except UnknownPromptVersion as exc: + raise click.ClickException(str(exc)) + except (ImportError, TypeError, ValueError) as exc: + raise click.ClickException(f"could not load adjudicator {path!r}: {exc}") + + if adjudicator.prompt_version != prompt_version: + raise click.ClickException( + f"the adjudicator loaded from {path!r} calls itself " + f"{adjudicator.prompt_version!r}, but --prompt-version says " + f"{prompt_version!r}. Refusing to label its output with a name it does " + f"not answer to." + ) + return adjudicator + + +@click.command(name="replay") +@click.option( + "--prompt-version", + required=True, + help="The adjudication prompt version to run and to label its output with.", +) +@click.option( + "--against", + default=None, + help="Diff against this version's stored evidence instead of --prompt-version's.", +) +@click.option( + "--adjudicator", + "adjudicator_path", + default=None, + help="Load 'module:attr' instead of the registered adjudicator for this version.", +) +@click.option("--limit", type=int, default=None, help="Replay only the first N observations.") +@click.option( + "--commit", + "do_commit", + is_flag=True, + default=False, + help="Persist the replayed evidence. Without this, nothing is written.", +) +def replay_command(prompt_version, against, adjudicator_path, limit, do_commit): + """Rebuild Evidence from Observations and print the diff. Writes only with --commit.""" + baseline_version = against or prompt_version + adjudicator = _adjudicator_for(prompt_version, adjudicator_path) + + with get_db() as session: + observations = session.query(Observation).count() + watches = session.query(Watch).count() + + if observations == 0: + click.echo( + warning( + "No observations are stored, so there is nothing to replay. " + "Evidence is rebuilt from observations alone; the 55,249 legacy " + "'articles' rows are not observations. See src/database/models.py." + ) + ) + return + + replayed = rebuild(session, adjudicator, limit=limit) + baseline = stored_evidence(session, baseline_version) + result = diff(replayed, baseline) + + click.echo(header("REPLAY")) + click.echo(format_diff(result, prompt_version, baseline_version, observations, watches)) + click.echo() + + if not do_commit: + click.echo( + muted( + "Nothing was written. Re-run with " + + accent("--commit") + + muted(" to persist this version's evidence.") + ) + ) + return + + try: + written = commit(session, prompt_version, replayed) + except NondeterministicReplay as exc: + raise click.ClickException(str(exc)) + + click.echo( + f"committed {prompt_version}: {written['inserted']} inserted, " + f"{written['deleted']} deleted" + ) diff --git a/src/config/settings.py b/src/config/settings.py index fe5d30f..568d6d4 100644 --- a/src/config/settings.py +++ b/src/config/settings.py @@ -20,6 +20,22 @@ class Settings(BaseSettings): os.getenv("SMART_RSS_FETCH_THRESHOLD_MINUTES", "60") ) + # How similar two observations' MinHash signatures must be before they are + # treated as the same item published twice. 0.7 was chosen against measured + # values rather than picked: on the two real corpus pairs in + # tests/sources/fixtures/near_duplicates.json, a genuine near-duplicate (the + # same press release carried by two Prince William feeds, differing only in + # byline and footer) scores 0.89, and the hardest distinct pair -- two + # different articles from the same publisher about the same county clerk's + # wedding events -- scores 0.016. 0.7 sits in the middle of that gap with + # room on both sides for text this suite has not seen. + # + # This is the *only* near-duplicate parameter that belongs in config. The + # signature width and shingle size are format, not preference; changing them + # silently invalidates every signature already stored, which is why they are + # constants in src/sources/minhash.py. (2026-08-31, backlog task 014.) + near_duplicate_threshold: float = float(os.getenv("NEAR_DUPLICATE_THRESHOLD", "0.7")) + project_root: Path = Path(__file__).parent.parent.parent data_dir: Path = project_root / "data" logs_dir: Path = project_root / "src" / "logs" diff --git a/src/database/migrations/add_observations_and_evidence.py b/src/database/migrations/add_observations_and_evidence.py new file mode 100644 index 0000000..89eae2c --- /dev/null +++ b/src/database/migrations/add_observations_and_evidence.py @@ -0,0 +1,110 @@ +""" +Migration: add the ``observations`` and ``evidence`` tables. + +Additive. It creates two tables and touches nothing that exists. In particular +it does not read, rewrite or migrate ``articles`` -- the 55,249 pre-rewrite rows +are left exactly as they are, and the standing rule for which table is +authoritative is written in ``src/database/models.py``. + +The DDL is not written out here. Both tables are created from their models, as +``add_watches_table`` does, so the migration and the models cannot drift. That +matters more than usual for ``observations``: its immutability trigger is +attached to the table's ``after_create`` event in the models module, so creating +the table from the model is what installs the trigger. Hand-written DDL here +would produce a table that looks right and silently accepts UPDATEs. + +Order matters on the way up and on the way down. ``evidence`` has foreign keys +into ``observations`` and ``watches``, so observations is created first and +dropped last. + +``downgrade()`` destroys data and requires ``--confirm``. Dropping ``evidence`` +loses adjudication output, which is derived and rebuildable by replay -- but +only if the observations that produced it still exist, which is exactly what +dropping ``observations`` in the same breath would take away. + +Added 2026-08-31 for backlog task 014. +""" + +from __future__ import annotations + +import argparse +import sys +from typing import Any + +from sqlalchemy import inspect +from sqlalchemy.engine import Engine + +from src.database.connection import engine +from src.database.models import Evidence, Observation + +__all__ = ["TABLES", "downgrade", "upgrade"] + +# Creation order. Reversed on the way down. +TABLES = ("observations", "evidence") + +# Annotated as Any because ``src.database.models`` is excluded from mypy (the +# pre-2.0 Column() style makes every attribute read as its column type), so the +# declarative classes arrive here as ``type[object]``. +_MODELS: dict[str, Any] = {"observations": Observation, "evidence": Evidence} + +_CONFIRM_HELP = ( + "Dropping these tables discards the content-addressed corpus and every\n" + "adjudication verdict recorded against it. Evidence is derived and can be\n" + "replayed -- but only from observations, which this also drops.\n" + "Re-run with --confirm if that is what you want:\n\n" + " python -m src.database.migrations.add_observations_and_evidence --down --confirm\n" +) + + +def upgrade(target: Engine | None = None) -> list[str]: + """Create whichever of the two tables is absent. Returns the names created.""" + target = target or engine + inspector = inspect(target) + created = [] + for name in TABLES: + if inspector.has_table(name): + print(f"{name} already exists; nothing to do.") + continue + model = _MODELS[name] + model.__table__.create(bind=target) + print(f"Created {name} ({len(model.__table__.columns)} columns).") + created.append(name) + return created + + +def downgrade(target: Engine | None = None, *, confirmed: bool = False) -> list[str]: + """Drop both tables. Requires ``confirmed`` -- this loses rows.""" + if not confirmed: + raise SystemExit(_CONFIRM_HELP) + target = target or engine + inspector = inspect(target) + dropped = [] + for name in reversed(TABLES): + if not inspector.has_table(name): + print(f"{name} does not exist; nothing to do.") + continue + _MODELS[name].__table__.drop(bind=target) + print(f"Dropped {name}.") + dropped.append(name) + return dropped + + +def main(argv: list[str] | None = None) -> int: + parser = argparse.ArgumentParser( + prog="add_observations_and_evidence", + description=__doc__, + formatter_class=argparse.RawDescriptionHelpFormatter, + ) + parser.add_argument("--down", action="store_true", help="drop the tables again") + parser.add_argument("--confirm", action="store_true", help="required by --down") + args = parser.parse_args(argv) + + if args.down: + downgrade(confirmed=args.confirm) + return 0 + upgrade() + return 0 + + +if __name__ == "__main__": + sys.exit(main()) diff --git a/src/database/models.py b/src/database/models.py index 357ac4b..fd2294b 100644 --- a/src/database/models.py +++ b/src/database/models.py @@ -11,15 +11,38 @@ removed here so that ``create_tables()`` cannot quietly recreate a concept the rewrite removed. -``articles`` is deliberately untouched. It holds the corpus (55,249 rows as of -2026-08-31) and what becomes of it is backlog task 014's decision, not this -one's. - ``watches`` is the first table of the rewrite, added 2026-08-31 by backlog task 013. Its CHECK constraints are not decoration -- see the class docstring. + +``observations`` and ``evidence`` are the second and third, added 2026-08-31 by +backlog task 014. + +**The rule for ``articles`` versus ``observations``, decided by task 014 and +written here because this is the file both of them live in.** They coexist, with +one direction of authority and one writer: + +* ``articles`` is the *legacy ingestion table*. It holds the 55,249 rows written + before the rewrite and it stays the row shape the pre-rewrite code reads. It + is not deleted and not migrated. +* ``observations`` is *authoritative for everything the rewrite builds*. A tier + added from task 014 onwards reads ``observations`` and never ``articles``. +* Exactly one code path writes an observation: + ``src.sources.observation.store_observation``, called from + ``src.sources.store.store_items``, which is the store path every adapter in + ``src/sources/`` already goes through. Each new article gets an observation in + the same transaction, and ``observations.article_id`` links the two. +* The 55,249 pre-existing rows have no observation, and neither does anything + the legacy ``src/rss/fetcher.py`` path writes directly. That is a known, + bounded gap, not an ambiguity: the content hash is a pure function of columns + those rows already carry, so a backfill is mechanical and is deliberately left + to its own task rather than run inside this one. + +The one thing that was not acceptable was two corpora with no stated rule. The +rule is: **new tiers read observations; articles is the pre-rewrite archive.** """ from sqlalchemy import ( + DDL, JSON, Boolean, CheckConstraint, @@ -33,6 +56,7 @@ String, Text, UniqueConstraint, + event, ) from sqlalchemy.ext.declarative import declarative_base from sqlalchemy.orm import relationship @@ -42,6 +66,18 @@ Base = declarative_base() +class ObservationIsImmutable(RuntimeError): + """ + Raised when anything tries to change a stored observation. + + Not a warning and not a silently-ignored write. An observation is + content-addressed, so its hash is a claim that the payload beside it is the + payload that produced the hash. A single successful UPDATE turns every + replay run afterwards into a comparison against a corpus that no longer + matches its own identities, and nothing would say so. + """ + + class RSSFeed(Base): """RSS feed sources - unchanged""" @@ -178,3 +214,171 @@ class Watch(Base): Index("idx_watches_decision_key", "decision_key"), Index("idx_watches_expires", "expires"), ) + + +class Observation(Base): + """ + One thing a source published, keyed by a hash of what it said. + + **The hash is the identity.** There is no surrogate primary key, because a + surrogate key would let the same document be stored twice under two ids and + the duplicate would be invisible. ``content_hash`` is computed by + ``src.sources.observation.observation_hash`` from the fields in + ``OBSERVATION_FIELDS`` and nothing else -- specifically not from + ``observed_at``, not from the source's last-fetch time, and not from + anything else that changes between two fetches of an unchanged document. + That exclusion is the whole invariant, so it is tested directly + (tests/sources/test_observation.py::TestHashIsContentOnly) rather than + inferred from the field list. + + ``observed_at`` is a column precisely *because* it is not in the hash: when + we first saw this content is worth keeping and is not part of what the + content is. It is the one per-fetch value on the row and it lives outside + the payload. + + ``payload`` is the normalized adapter output, verbatim, exactly the values + that were hashed. It is never rewritten -- see + :class:`ObservationIsImmutable` and the BEFORE UPDATE trigger below. + + ``minhash`` is the near-duplicate signature of the item's text, written once + at insert. It is stored rather than recomputed so that a grouping computed + now and a grouping computed after a replay are the same grouping. See + ``src/sources/minhash.py``. + + Added 2026-08-31 for backlog task 014. + """ + + __tablename__ = "observations" + + content_hash = Column(String(80), primary_key=True) + + # Which source published it. The same text from two sources is two + # observations, on purpose: "who said this" is part of what was observed, + # and grouping the pair back together is what the MinHash signature is for. + source_id = Column(Integer, ForeignKey("rss_feeds.id"), nullable=False) + + # The legacy row this observation was written alongside. Nullable because a + # backfill from the pre-rewrite corpus would set it and a future + # observations-only source may not have one. See the module docstring's rule. + article_id = Column(Integer, ForeignKey("articles.id")) + + payload = Column(JSON, nullable=False) + minhash = Column(JSON, nullable=False) + + # A projection of payload["published_date"], written by the same call that + # writes the payload, so that "observations in this window" is an indexed + # query rather than a JSON scan. Immutability is what keeps it honest: it + # cannot drift from the payload because neither can change. + published_date = Column(DateTime) + + observed_at = Column(DateTime, default=utcnow, nullable=False) + + __table_args__ = ( + CheckConstraint("content_hash LIKE 'sha256:%'", name="ck_observations_hash_prefix"), + Index("idx_observations_source", "source_id"), + Index("idx_observations_published", "published_date"), + Index("idx_observations_article", "article_id"), + ) + + +# Immutability, enforced twice on purpose. +# +# 1. The ORM guard below stops a mutation made through a Session, which is how +# application code would do it, and raises a named Python exception. +# 2. The trigger stops an UPDATE issued any other way -- a raw +# `session.execute(text("UPDATE ..."))`, the sqlite3 shell, a future +# migration written in a hurry. A guarantee that only holds for code that +# goes through the ORM is a convention, not a guarantee. +# +# The trigger is attached to the table's after_create, so it exists both when +# `Base.metadata.create_all()` builds a test database and when +# `src.database.migrations.add_observations_and_evidence` creates the real one. +# There is no second copy of the DDL to forget to update. +_OBSERVATIONS_IMMUTABLE_TRIGGER = DDL( + "CREATE TRIGGER observations_are_immutable " + "BEFORE UPDATE ON observations " + "BEGIN " + "SELECT RAISE(ABORT, " + "'observations are immutable: the content hash is the identity, so changing " + "a stored row would leave its hash describing content that is no longer there. " + "Insert a new observation instead.'); " + "END" +) + +event.listen( + Observation.__table__, + "after_create", + _OBSERVATIONS_IMMUTABLE_TRIGGER.execute_if(dialect="sqlite"), +) + + +@event.listens_for(Observation, "before_update", propagate=True) +def _refuse_observation_update(_mapper, _connection, target): + """Refuse an ORM flush that would UPDATE an observation.""" + raise ObservationIsImmutable( + f"refusing to update observation {target.content_hash}: observations are " + f"immutable once written. If the content changed, it is a different " + f"observation with a different hash." + ) + + +class Evidence(Base): + """ + One adjudicated link from an observation to a watch, under a named prompt. + + **``prompt_version`` is on every row, and that is the point of the table.** + Adjudication is the only stochastic tier in the system; the only way to tell + an improvement from a regression is to hold the observations fixed, run a + different prompt version over them, and diff. That requires knowing which + version produced which row, per row -- a version recorded once per run, in a + log, or in a config file is a version that cannot answer "and where did this + row come from" six weeks later. + + The uniqueness constraint ``(observation_hash, watch_id, prompt_version)`` is + what makes replay idempotent: one verdict per observation per watch per + prompt version, so re-running a version either inserts nothing or reveals + that the version is not deterministic. + + ``direction`` is two-valued. There is no 'neutral': an observation that + neither supports nor contradicts a watch produces no row, because a table of + non-evidence is a table nobody can read. + + Evidence is *derived*. It is the one thing here that may be deleted and + rebuilt, and ``insightweaver replay --commit`` does exactly that, scoped to a + single prompt version. + + Added 2026-08-31 for backlog task 014. + """ + + __tablename__ = "evidence" + + id = Column(Integer, primary_key=True) + + observation_hash = Column(String(80), ForeignKey("observations.content_hash"), nullable=False) + watch_id = Column(String(100), ForeignKey("watches.id"), nullable=False) + + direction = Column(String(20), nullable=False) + magnitude = Column(Float, nullable=False) + + prompt_version = Column(String(100), nullable=False) + rationale = Column(Text) + + created_at = Column(DateTime, default=utcnow) + + __table_args__ = ( + UniqueConstraint( + "observation_hash", + "watch_id", + "prompt_version", + name="_evidence_observation_watch_version_uc", + ), + CheckConstraint("direction IN ('supports', 'contradicts')", name="ck_evidence_direction"), + CheckConstraint( + "magnitude >= 0.0 AND magnitude <= 1.0", name="ck_evidence_magnitude_range" + ), + CheckConstraint( + "length(trim(prompt_version)) > 0", name="ck_evidence_prompt_version_present" + ), + Index("idx_evidence_prompt_version", "prompt_version"), + Index("idx_evidence_watch", "watch_id"), + ) diff --git a/src/evidence/__init__.py b/src/evidence/__init__.py new file mode 100644 index 0000000..83c3b9a --- /dev/null +++ b/src/evidence/__init__.py @@ -0,0 +1,56 @@ +""" +Evidence: what an adjudication prompt concluded, and the harness that replays it. + +Two modules and a hard line between them. ``adjudicator`` is the seam a prompt +version plugs into; ``replay`` rebuilds evidence from observations through that +seam and diffs the result against what is stored. Neither contains an +adjudication prompt -- that is backlog task 016. + +Added 2026-08-31 for backlog task 014. +""" + +from .adjudicator import ( + DIRECTIONS, + NULL_PROMPT_VERSION, + Adjudicator, + NullAdjudicator, + ObservationView, + UnknownPromptVersion, + Verdict, + known_prompt_versions, + load_adjudicator, + register, + resolve, +) +from .replay import ( + EvidenceRow, + NondeterministicReplay, + ReplayDiff, + commit, + diff, + format_diff, + rebuild, + stored_evidence, +) + +__all__ = [ + "DIRECTIONS", + "NULL_PROMPT_VERSION", + "Adjudicator", + "EvidenceRow", + "NondeterministicReplay", + "NullAdjudicator", + "ObservationView", + "ReplayDiff", + "UnknownPromptVersion", + "Verdict", + "commit", + "diff", + "format_diff", + "known_prompt_versions", + "load_adjudicator", + "rebuild", + "register", + "resolve", + "stored_evidence", +] diff --git a/src/evidence/adjudicator.py b/src/evidence/adjudicator.py new file mode 100644 index 0000000..54feeb6 --- /dev/null +++ b/src/evidence/adjudicator.py @@ -0,0 +1,210 @@ +""" +The seam a prompt version plugs into, and the vocabulary it may return. + +Adjudication is the one stochastic tier in this system. Everything either side +of it is deterministic and unit-testable; the tier itself can only be reviewed +by holding its inputs fixed and looking at what a different prompt does to its +outputs. This module is the shape of that seam. **It does not contain the +adjudication prompt** -- that is backlog task 016, and deliberately not here. + +Three properties are enforced rather than documented: + +1. **An adjudicator sees observations and watches, nothing else.** The argument + is an :class:`ObservationView`, which carries a content hash and the stored + payload -- not an ORM row, not a database session, not the clock. That is + what makes "Evidence is a pure function of (Observations, prompt version)" + checkable instead of aspirational. +2. **An adjudicator names its own prompt version**, and the replay harness + writes that name onto every row it produces. A version that came from a + command-line flag while the model came from somewhere else is how a + mislabelled corpus happens. +3. **A verdict is validated on construction.** Direction is two-valued and + magnitude is bounded; a bad one raises where it was built rather than at the + CHECK constraint several layers later, though the CHECK constraint is there + too. + +**On the shipped adjudicator.** :class:`NullAdjudicator` returns no verdicts. +It exists so the harness is runnable and testable before task 016 supplies +something real, and it is honest: an adjudicator that guessed a direction from a +keyword match would put fabricated judgements in a table whose entire purpose is +to make judgements reviewable. Registering a real one is +:func:`register`; running an unregistered one for a one-off is +``--adjudicator module:attr`` on the CLI. + +Added 2026-08-31 for backlog task 014. +""" + +from __future__ import annotations + +import importlib +from collections.abc import Callable, Sequence +from dataclasses import dataclass, field +from typing import Any, Protocol, runtime_checkable + +__all__ = [ + "Adjudicator", + "DIRECTIONS", + "NULL_PROMPT_VERSION", + "NullAdjudicator", + "ObservationView", + "UnknownPromptVersion", + "Verdict", + "known_prompt_versions", + "load_adjudicator", + "register", + "resolve", +] + +# Two values, no 'neutral'. See src/database/models.py::Evidence. +DIRECTIONS: tuple[str, ...] = ("supports", "contradicts") + +NULL_PROMPT_VERSION = "null-v0" + + +@dataclass(frozen=True) +class ObservationView: + """ + Everything an adjudicator is allowed to see about one observation. + + The payload is the stored payload, verbatim. Nothing derived, nothing + fetched, no session. + """ + + content_hash: str + payload: dict[str, Any] + + @property + def title(self) -> str: + return str(self.payload.get("title") or "") + + @property + def text(self) -> str: + return str(self.payload.get("normalized_content") or self.payload.get("description") or "") + + +@dataclass(frozen=True) +class Verdict: + """One adjudicated link from the observation to a watch.""" + + watch_id: str + direction: str + magnitude: float + rationale: str = field(default="") + + def __post_init__(self) -> None: + if self.direction not in DIRECTIONS: + raise ValueError( + f"verdict direction must be one of {DIRECTIONS}, got {self.direction!r}" + ) + if not 0.0 <= self.magnitude <= 1.0: + raise ValueError(f"verdict magnitude must be in [0, 1], got {self.magnitude}") + if not str(self.watch_id).strip(): + raise ValueError("a verdict must name the watch it bears on") + + +@runtime_checkable +class Adjudicator(Protocol): + """What a prompt version has to look like to be replayable.""" + + prompt_version: str + + def adjudicate(self, observation: ObservationView, watches: Sequence[Any]) -> list[Verdict]: + """ + The verdicts this observation supports, for the watches given. + + Must be a pure function of its arguments. Returning ``[]`` is normal and + is the common case -- most observations bear on no watch. + """ + ... + + +class NullAdjudicator: + """ + The adjudicator that finds nothing. See the module docstring. + + Useful for more than bootstrapping: replaying it against a stored prompt + version shows every row that version produced, as a removal, which is a + quick way to see what a version is actually claiming. + """ + + prompt_version = NULL_PROMPT_VERSION + + def adjudicate(self, observation: ObservationView, watches: Sequence[Any]) -> list[Verdict]: # noqa: ARG002 + return [] + + +class UnknownPromptVersion(KeyError): + """Raised when no adjudicator is registered for a requested prompt version.""" + + def __init__(self, version: str, known: Sequence[str]) -> None: + self.version = version + self.known = list(known) + super().__init__( + f"no adjudicator is registered for prompt version {version!r} " + f"(registered: {', '.join(self.known) or 'none'}). Register one with " + f"src.evidence.adjudicator.register(), or pass " + f"--adjudicator module:attr to name one directly." + ) + + def __str__(self) -> str: + return self.args[0] + + +_REGISTRY: dict[str, Callable[[], Adjudicator]] = { + NULL_PROMPT_VERSION: NullAdjudicator, +} + + +def register(version: str, factory: Callable[[], Adjudicator]) -> None: + """ + Bind a prompt version to the adjudicator that implements it. + + Re-registering a version is refused. Two builds of "v3" that disagree would + make every stored ``prompt_version`` ambiguous, which is the one thing the + column exists to prevent. + """ + if version in _REGISTRY: + raise ValueError( + f"prompt version {version!r} is already registered; a version identifies " + f"one adjudicator, and rebinding it would make stored evidence ambiguous" + ) + _REGISTRY[version] = factory + + +def known_prompt_versions() -> list[str]: + """Every registered prompt version, sorted.""" + return sorted(_REGISTRY) + + +def resolve(version: str) -> Adjudicator: + """The adjudicator for ``version``, or :class:`UnknownPromptVersion`.""" + factory = _REGISTRY.get(version) + if factory is None: + raise UnknownPromptVersion(version, known_prompt_versions()) + return factory() + + +def load_adjudicator(path: str) -> Adjudicator: + """ + Import ``module:attr`` and call it, for an adjudicator not in the registry. + + The escape hatch for a prompt version under development and for the test + suite's stubs. It is explicit on the command line rather than read from the + environment, so that a replay whose adjudicator did not come from the + registry says so in the shell history. + """ + if ":" not in path: + raise ValueError(f"adjudicator path must be 'module:attr', got {path!r}") + module_name, _, attr = path.partition(":") + module = importlib.import_module(module_name) + try: + factory = getattr(module, attr) + except AttributeError: + raise ValueError(f"{module_name} has no attribute {attr!r}") + adjudicator = factory() if callable(factory) else factory + if not isinstance(adjudicator, Adjudicator): + raise TypeError( + f"{path} did not produce an Adjudicator " + f"(needs a 'prompt_version' attribute and an 'adjudicate' method)" + ) + return adjudicator diff --git a/src/evidence/replay.py b/src/evidence/replay.py new file mode 100644 index 0000000..4b8f88d --- /dev/null +++ b/src/evidence/replay.py @@ -0,0 +1,282 @@ +""" +Rebuilding Evidence from Observations, and diffing it against what is stored. + +This is the mechanism the whole task exists for. Observations are immutable and +content-addressed; Evidence is a pure function of (Observations, prompt +version). Given both, "what did this prompt change do" stops being a matter of +opinion and becomes a diff. + +**Reading order matters and is fixed.** Observations are read in content-hash +order, watches in id order, and the output is sorted by ``(observation, watch)``. +A replay whose output order depended on a query plan would produce a different +byte stream on a database with different statistics, and the reproducibility +claim would be false in a way nobody would notice. + +**What is compared against what.** ``--prompt-version X`` computes X's output. +The baseline defaults to the stored rows *for X*, which makes a plain replay a +reproducibility check: a deterministic adjudicator must produce an empty diff. +``--against Y`` compares X's output to Y's stored rows instead, which is the +"what did my prompt change do" view. + +**Committing is separate from diffing, and narrower.** :func:`commit` writes +only for the target prompt version, and it refuses outright if the target +version already has a row for a key with a different verdict -- that means the +same version produced two different answers, and quietly overwriting the first +one would erase the evidence of the very problem the harness exists to surface. + +Added 2026-08-31 for backlog task 014. +""" + +from __future__ import annotations + +import logging +from collections.abc import Sequence +from dataclasses import dataclass, field + +from sqlalchemy.orm import Session + +from ..database.models import Evidence, Observation, Watch +from .adjudicator import Adjudicator, ObservationView + +logger = logging.getLogger(__name__) + +__all__ = [ + "EvidenceRow", + "NondeterministicReplay", + "ReplayDiff", + "commit", + "diff", + "format_diff", + "rebuild", + "stored_evidence", +] + + +@dataclass(frozen=True, order=True) +class EvidenceRow: + """One evidence verdict, without its surrogate id or its clock.""" + + observation_hash: str + watch_id: str + direction: str + magnitude: float + rationale: str = "" + + @property + def key(self) -> tuple[str, str]: + return (self.observation_hash, self.watch_id) + + def describe(self) -> str: + suffix = f" {self.rationale}" if self.rationale else "" + return f"{self.direction} {self.magnitude:.3f}{suffix}" + + +class NondeterministicReplay(RuntimeError): + """ + Raised when a prompt version produces a verdict differing from its own + stored row. + + This is the failure the harness is built to catch, so it is loud and it + stops the commit. See :func:`commit`. + """ + + +@dataclass +class ReplayDiff: + """What changed between a replay's output and a stored baseline.""" + + added: list[EvidenceRow] = field(default_factory=list) + removed: list[EvidenceRow] = field(default_factory=list) + changed: list[tuple[EvidenceRow, EvidenceRow]] = field(default_factory=list) + + @property + def is_empty(self) -> bool: + return not (self.added or self.removed or self.changed) + + @property + def counts(self) -> dict[str, int]: + return { + "added": len(self.added), + "removed": len(self.removed), + "changed": len(self.changed), + } + + +def rebuild( + db: Session, + adjudicator: Adjudicator, + limit: int | None = None, +) -> list[EvidenceRow]: + """ + Run ``adjudicator`` over the stored observations and return what it produced. + + Reads observations and watches. Writes nothing, ever -- committing is + :func:`commit`, and keeping them separate is what makes a diff safe to run + against a corpus you care about. + """ + watches = db.query(Watch).order_by(Watch.id).all() + query = db.query(Observation.content_hash, Observation.payload).order_by( + Observation.content_hash + ) + if limit is not None: + query = query.limit(limit) + + rows: list[EvidenceRow] = [] + for content_hash, payload in query.all(): + view = ObservationView(content_hash=content_hash, payload=dict(payload or {})) + for verdict in adjudicator.adjudicate(view, watches): + rows.append( + EvidenceRow( + observation_hash=content_hash, + watch_id=verdict.watch_id, + direction=verdict.direction, + magnitude=verdict.magnitude, + rationale=verdict.rationale, + ) + ) + return sorted(rows) + + +def stored_evidence(db: Session, prompt_version: str) -> list[EvidenceRow]: + """The evidence already stored for one prompt version, in the same order.""" + rows = ( + db.query( + Evidence.observation_hash, + Evidence.watch_id, + Evidence.direction, + Evidence.magnitude, + Evidence.rationale, + ) + .filter(Evidence.prompt_version == prompt_version) + .all() + ) + return sorted(EvidenceRow(row[0], row[1], row[2], float(row[3]), row[4] or "") for row in rows) + + +def diff(replayed: Sequence[EvidenceRow], baseline: Sequence[EvidenceRow]) -> ReplayDiff: + """ + What ``replayed`` says that ``baseline`` does not, keyed by (observation, watch). + + A key present in both with a different verdict is ``changed`` rather than an + add and a remove, because the interesting question about a prompt change is + which judgements it revised, not how many rows moved. + """ + left = {row.key: row for row in replayed} + right = {row.key: row for row in baseline} + + result = ReplayDiff() + for key in sorted(left.keys() | right.keys()): + new, old = left.get(key), right.get(key) + if old is None and new is not None: + result.added.append(new) + elif new is None and old is not None: + result.removed.append(old) + elif new != old and new is not None and old is not None: + result.changed.append((old, new)) + return result + + +def commit( + db: Session, + prompt_version: str, + replayed: Sequence[EvidenceRow], +) -> dict[str, int]: + """ + Make the stored evidence for ``prompt_version`` equal ``replayed``. + + Inserts what is missing and deletes what the replay did not produce, both + scoped to this one prompt version -- no other version's rows are read or + touched. Evidence is derived, so deleting a derived row that the current + definition no longer produces is correct; observations are not derived, and + nothing here can touch one. + + Refuses, before writing anything, if a key exists for this version with a + different verdict. See :class:`NondeterministicReplay`. + """ + current = stored_evidence(db, prompt_version) + against_self = diff(replayed, current) + + if against_self.changed: + detail = "\n".join( + f" {old.observation_hash} / {old.watch_id}: " + f"stored [{old.describe()}] -> replayed [{new.describe()}]" + for old, new in against_self.changed[:10] + ) + raise NondeterministicReplay( + f"prompt version {prompt_version!r} produced {len(against_self.changed)} " + f"verdict(s) that differ from its own stored rows. Evidence is supposed to " + f"be a pure function of (observations, prompt version), so this means either " + f"the adjudicator is not deterministic or the version label is being reused " + f"for two different adjudicators. Nothing was written.\n{detail}" + ) + + for row in against_self.added: + db.add( + Evidence( + observation_hash=row.observation_hash, + watch_id=row.watch_id, + direction=row.direction, + magnitude=row.magnitude, + prompt_version=prompt_version, + rationale=row.rationale or None, + ) + ) + + deleted = 0 + for row in against_self.removed: + deleted += ( + db.query(Evidence) + .filter( + Evidence.prompt_version == prompt_version, + Evidence.observation_hash == row.observation_hash, + Evidence.watch_id == row.watch_id, + ) + .delete(synchronize_session=False) + ) + db.flush() + + written = {"inserted": len(against_self.added), "deleted": deleted} + logger.info( + f"replay --commit {prompt_version}: inserted {written['inserted']}, " + f"deleted {written['deleted']}" + ) + return written + + +def format_diff( + result: ReplayDiff, + prompt_version: str, + baseline_version: str, + observations: int, + watches: int, +) -> str: + """ + The diff as text, deterministic to the byte. + + No timestamps, no row ids, no counts of anything that varies between two + runs over the same corpus -- a report that changed between identical runs + could not be used to prove that the runs were identical. + """ + lines = [ + f"replay: prompt-version={prompt_version} against={baseline_version}", + f"corpus: {observations} observation(s), {watches} watch(es)", + ( + f"diff: {result.counts['added']} added, " + f"{result.counts['removed']} removed, " + f"{result.counts['changed']} changed" + ), + ] + if result.is_empty: + lines.append("") + lines.append("No difference from the stored evidence.") + return "\n".join(lines) + + for row in result.added: + lines.append(f"+ {row.observation_hash} / {row.watch_id}: {row.describe()}") + for row in result.removed: + lines.append(f"- {row.observation_hash} / {row.watch_id}: {row.describe()}") + for old, new in result.changed: + lines.append(f"~ {old.observation_hash} / {old.watch_id}:") + lines.append(f" was [{old.describe()}]") + lines.append(f" now [{new.describe()}]") + return "\n".join(lines) diff --git a/src/sources/__init__.py b/src/sources/__init__.py index 55022ff..3392b2b 100644 --- a/src/sources/__init__.py +++ b/src/sources/__init__.py @@ -8,22 +8,38 @@ (deduplication, filtering, clustering, frames, questions, predictions, synthesis) is untouched and has no idea an adapter exists. +Amended 2026-08-31 (backlog task 014): the same store path now also writes an +immutable, content-addressed :class:`~src.database.models.Observation` for every +item, through :mod:`src.sources.observation` and nowhere else. ``minhash`` +carries the near-duplicate signature written beside it. + See ``SOURCES.md`` at the repository root for the recorded basis of use for each configured source. """ from .base import ARTICLE_FIELDS, RawItem, SourceAdapter, SourceUnavailable, content_hash from .federal_register import FederalRegisterAdapter, FederalRegisterFilter, FederalRegisterQuery +from .minhash import group_near_duplicates, signature, similarity +from .observation import ( + OBSERVATION_FIELDS, + ObservationRecord, + near_duplicate_groups, + observation_hash, + observe, + store_observation, +) from .rss_adapter import RSSAdapter from .runner import IngestResult, build_configured_adapters, run_adapter, run_configured_adapters from .store import ensure_source, store_items __all__ = [ "ARTICLE_FIELDS", + "OBSERVATION_FIELDS", "FederalRegisterAdapter", "FederalRegisterFilter", "FederalRegisterQuery", "IngestResult", + "ObservationRecord", "RSSAdapter", "RawItem", "SourceAdapter", @@ -31,7 +47,14 @@ "build_configured_adapters", "content_hash", "ensure_source", + "group_near_duplicates", + "near_duplicate_groups", + "observation_hash", + "observe", "run_adapter", "run_configured_adapters", + "signature", + "similarity", "store_items", + "store_observation", ] diff --git a/src/sources/minhash.py b/src/sources/minhash.py new file mode 100644 index 0000000..3af831c --- /dev/null +++ b/src/sources/minhash.py @@ -0,0 +1,191 @@ +""" +Near-duplicate detection by MinHash over word shingles. + +**What this is for.** Two feeds carry the same wire story with a different +byline, a different photo credit and a different "appeared first on" footer. +Their content hashes differ -- correctly, because the bytes differ -- so +content-addressing cannot group them. Something has to, or the same event is +adjudicated three times and counts as three pieces of evidence for a Watch. +That is the failure this module exists to prevent. + +**Why MinHash and not the existing similarity code.** +``src/processors/deduplicator.py`` compares articles pairwise with a word-set +Jaccard computed from scratch on every comparison, inside a time window, at +query time. It is not wrong, but nothing about it is storable: there is no +per-article artefact you can keep and compare later, so the grouping cannot be +recomputed identically after the fact. A MinHash signature *is* that artefact. +It is a fixed-width function of the text alone, written once beside the +observation, and the similarity of two observations is then a comparison of two +short integer vectors -- so a grouping computed today and a grouping computed +after a replay agree by construction. + +**The parameters are format, not preference.** +:data:`PERMUTATIONS` and :data:`SHINGLE_SIZE` determine what a stored signature +*means*. Change either and every signature already written becomes +incomparable with every signature written afterwards, silently -- the vectors +are still the same width, the numbers are still numbers, and the similarities +are just wrong. So they are module constants with this warning attached, not +configuration. The *threshold* is the tunable one, because it is applied at +comparison time and changing it re-groups the same stored signatures +consistently; it lives in ``settings.near_duplicate_threshold``. + +The permutation coefficients are derived from BLAKE2b of the permutation index +rather than from ``random.Random(seed)``. Both are deterministic today, but only +one of them is deterministic because of a documented property of a hash function +rather than a documented property of an implementation's PRNG. + +Added 2026-08-31 for backlog task 014. +""" + +from __future__ import annotations + +import hashlib +from collections.abc import Mapping, Sequence + +from .base import normalize_for_hash + +__all__ = [ + "PERMUTATIONS", + "SHINGLE_SIZE", + "group_near_duplicates", + "shingles", + "signature", + "similarity", +] + +# Words per shingle. Five is long enough that two documents sharing a stock +# phrase ("the department said in a statement") do not look related on that +# alone, and short enough that a rewritten sentence in an otherwise identical +# document still leaves most shingles intact. +SHINGLE_SIZE = 5 + +# Signature width. 128 puts the standard error of the Jaccard estimate near +# 1/sqrt(128) ~= 0.09, which is far below the margin between the two cases this +# has to separate (see tests/sources/test_minhash.py: a real near-duplicate pair +# lands around 0.9, a real distinct pair around 0.0). +PERMUTATIONS = 128 + +# 2**61 - 1, a Mersenne prime: the modulus for the (a*x + b) mod p permutations. +_MODULUS = (1 << 61) - 1 + + +def _coefficients(index: int) -> tuple[int, int]: + """The (a, b) pair for permutation ``index``, derived from its number.""" + digest = hashlib.blake2b(f"insightweaver/minhash/{index}".encode(), digest_size=16).digest() + a = int.from_bytes(digest[:8], "big") % (_MODULUS - 1) + 1 + b = int.from_bytes(digest[8:], "big") % _MODULUS + return a, b + + +_COEFFICIENTS: tuple[tuple[int, int], ...] = tuple(_coefficients(i) for i in range(PERMUTATIONS)) + + +def shingles(text: str) -> set[int]: + """ + The hashed word-shingles of ``text``. + + Normalization is :func:`src.sources.base.normalize_for_hash` -- the same + function the content hash uses -- so "the same text" means one thing in this + repository rather than two. + + A document shorter than :data:`SHINGLE_SIZE` words yields one shingle for + the whole document rather than none, so that short items are comparable with + each other instead of silently matching nothing. + """ + words = normalize_for_hash(text).split() + if not words: + return set() + if len(words) <= SHINGLE_SIZE: + grams = [" ".join(words)] + else: + grams = [ + " ".join(words[i : i + SHINGLE_SIZE]) for i in range(len(words) - SHINGLE_SIZE + 1) + ] + return { + int.from_bytes(hashlib.blake2b(g.encode("utf-8"), digest_size=8).digest(), "big") + for g in grams + } + + +def signature(text: str) -> tuple[int, ...]: + """ + The MinHash signature of ``text``: :data:`PERMUTATIONS` integers. + + Empty text has an empty signature rather than a signature of sentinels. + An empty signature is 0.0-similar to everything including another empty one, + which is the honest answer -- two items we know nothing about are not + evidence of the same event. + """ + hashed = shingles(text) + if not hashed: + return () + return tuple(min((a * h + b) % _MODULUS for h in hashed) for a, b in _COEFFICIENTS) + + +def similarity(left: Sequence[int], right: Sequence[int]) -> float: + """ + Estimated Jaccard similarity of two signatures: the fraction that agree. + + Signatures of different non-zero widths are a programming error, not a + similarity of zero -- it means signatures written under two different + :data:`PERMUTATIONS` values are being compared, which is the silent + corruption the module docstring warns about, so it raises. + """ + if not left or not right: + return 0.0 + if len(left) != len(right): + raise ValueError( + f"cannot compare MinHash signatures of width {len(left)} and {len(right)}: " + f"they were computed under different PERMUTATIONS settings and their " + f"similarity would be meaningless" + ) + return sum(1 for a, b in zip(left, right, strict=True) if a == b) / len(left) + + +def group_near_duplicates( + signatures: Mapping[str, Sequence[int]], + threshold: float, +) -> list[list[str]]: + """ + Group keys whose signatures are at least ``threshold`` similar. + + Returns every group, including singletons, sorted by key and then by first + key, so the output is a deterministic function of the input -- a grouping + that depended on dict order would break the replay guarantee it supports. + + Transitive by union-find: if A is near B and B is near C then all three land + in one group even if A and C fall just under the threshold. That is the + behaviour wanted for a wire story picked up in a chain of rewrites. + + Pairwise, so O(n^2) comparisons. That is deliberate at this size -- a + day of ingestion is a few hundred observations and the comparison is 128 + integer equalities. An LSH index is the answer if this is ever pointed at + the whole corpus, and it would change the grouping only by making it + approximate. + """ + if not 0.0 <= threshold <= 1.0: + raise ValueError(f"near-duplicate threshold must be in [0, 1], got {threshold}") + + keys = sorted(signatures) + parent = {key: key for key in keys} + + def find(key: str) -> str: + while parent[key] != key: + parent[key] = parent[parent[key]] + key = parent[key] + return key + + def union(a: str, b: str) -> None: + root_a, root_b = find(a), find(b) + if root_a != root_b: + parent[max(root_a, root_b)] = min(root_a, root_b) + + for i, left in enumerate(keys): + for right in keys[i + 1 :]: + if similarity(signatures[left], signatures[right]) >= threshold: + union(left, right) + + groups: dict[str, list[str]] = {} + for key in keys: + groups.setdefault(find(key), []).append(key) + return sorted((sorted(members) for members in groups.values()), key=lambda g: g[0]) diff --git a/src/sources/observation.py b/src/sources/observation.py new file mode 100644 index 0000000..0cd3c65 --- /dev/null +++ b/src/sources/observation.py @@ -0,0 +1,214 @@ +""" +The one path by which an adapter's output becomes a stored Observation. + +Every adapter in ``src/sources/`` reaches the database through +``src.sources.store.store_items``, and ``store_items`` reaches ``observations`` +through :func:`store_observation` here. There is deliberately no second +constructor, no bulk-insert shortcut and no "just this once" path -- the +guarantee that observations are content-addressed is only as strong as the +number of places that can write one. +tests/sources/test_observation.py::TestOneWritePath asserts that count against +the source tree rather than against anyone's intention. + +**What is hashed, and why the list is closed.** +:data:`OBSERVATION_FIELDS` is ``ARTICLE_FIELDS`` (which +tests/sources/test_base.py already pins to the live RSS normalizer) minus +``word_count``, plus ``source_url``. Everything in the payload is hashed and +nothing outside the payload is: + +* ``word_count`` is dropped because it is derived from ``normalized_content``. + Storing it would put a second copy of one fact in an immutable row. +* ``source_url`` is added because the same text published by two sources is two + observations. Who published it is part of what was observed. Grouping the + pair back together is :mod:`src.sources.minhash`'s job, not the hash's. +* Nothing per-fetch is in the list. This is the landmine the task named: a fetch + timestamp or a session id inside the hashed payload stores the same document + on every run, and the symptom looks like broken dedup rather than a broken + hash. ``observed_at`` is a column on the row and is not in the payload; + ``RSSFetcher.normalize_article`` leaves ``published_date`` as ``None`` when + the feed gives no date rather than defaulting it to now, which was checked + before this list was written (2026-08-31). + +The hash itself is :func:`src.sources.base.content_hash`, the function task 005 +already introduced. There is one hashing scheme in this repository. + +Added 2026-08-31 for backlog task 014. +""" + +from __future__ import annotations + +import logging +from dataclasses import dataclass +from datetime import datetime +from typing import Any + +from sqlalchemy.orm import Session + +from ..config.settings import settings +from ..database.models import Observation, RSSFeed +from .base import ARTICLE_FIELDS, RawItem, content_hash +from .minhash import group_near_duplicates, signature + +logger = logging.getLogger(__name__) + +__all__ = [ + "OBSERVATION_FIELDS", + "ObservationRecord", + "near_duplicate_groups", + "observation_hash", + "observation_payload", + "observe", + "store_observation", +] + +# The closed set of payload keys, in hash order. See the module docstring. +OBSERVATION_FIELDS: tuple[str, ...] = ("source_url",) + tuple( + field for field in ARTICLE_FIELDS if field != "word_count" +) + + +def _render(value: Any) -> str: + """One payload value as the string that goes into the hash.""" + if value is None: + return "" + if isinstance(value, datetime): + return value.isoformat() + if isinstance(value, list | tuple): + return " ".join(_render(item) for item in value) + return str(value) + + +def observation_payload(source_url: str, item: RawItem) -> dict[str, Any]: + """ + The normalized adapter output for ``item``, verbatim, as stored. + + ``published_date`` is rendered as an ISO string so that the payload is JSON + round-trippable: a payload that only survives as Python objects is not a raw + payload retained verbatim, it is a payload retained until the next reader. + """ + fields = item.as_article_fields() + fields["source_url"] = source_url + payload = {key: fields[key] for key in OBSERVATION_FIELDS} + published = payload.get("published_date") + payload["published_date"] = published.isoformat() if published is not None else None + return payload + + +def observation_hash(payload: dict[str, Any]) -> str: + """ + The content hash of a payload. A pure function of the payload's values. + + Raises on a payload whose keys are not exactly :data:`OBSERVATION_FIELDS`: + an extra key would be stored but unhashed, which is the quiet way a + content-addressed store stops being content-addressed. + """ + if set(payload) != set(OBSERVATION_FIELDS): + missing = sorted(set(OBSERVATION_FIELDS) - set(payload)) + extra = sorted(set(payload) - set(OBSERVATION_FIELDS)) + raise ValueError( + f"observation payload does not match OBSERVATION_FIELDS " + f"(missing: {missing}, unexpected: {extra}); every stored key must be hashed" + ) + return content_hash(*(_render(payload[key]) for key in OBSERVATION_FIELDS)) + + +def minhash_text(payload: dict[str, Any]) -> str: + """ + The text a near-duplicate signature is computed over: title plus body. + + ``normalized_content`` is preferred over ``description`` because it is the + HTML-stripped full text; ``description`` is the fallback for feeds that + carry only a summary. + """ + body = payload.get("normalized_content") or payload.get("description") or "" + return f"{payload.get('title') or ''} {body}".strip() + + +@dataclass(frozen=True) +class ObservationRecord: + """A hashed, signed observation, not yet stored.""" + + content_hash: str + payload: dict[str, Any] + minhash: tuple[int, ...] + published_date: datetime | None + + +def observe(source_url: str, item: RawItem) -> ObservationRecord: + """Normalize, hash and sign one adapter item. Touches no database.""" + payload = observation_payload(source_url, item) + return ObservationRecord( + content_hash=observation_hash(payload), + payload=payload, + minhash=signature(minhash_text(payload)), + published_date=item.published_date, + ) + + +def store_observation( + db: Session, + source: RSSFeed, + item: RawItem, + article_id: int | None = None, +) -> tuple[str, bool]: + """ + Store one observation if it is not already stored. Returns ``(hash, created)``. + + Re-seeing content we already hold is a no-op, not an update and not an + error: that is what content-addressing buys, and it is why re-running an + adapter over an unchanged upstream costs one request and zero writes. + + Note what this function cannot do. There is no argument by which a caller + supplies a hash, so the stored hash is always the hash of the stored payload; + and if the row exists it returns without touching it, so this path cannot be + the one that mutates an observation. Attempting it anyway raises + :class:`~src.database.models.ObservationIsImmutable`. + """ + record = observe(str(source.url), item) + + existing = ( + db.query(Observation.content_hash) + .filter(Observation.content_hash == record.content_hash) + .first() + ) + if existing is not None: + return record.content_hash, False + + db.add( + Observation( + content_hash=record.content_hash, + source_id=source.id, + article_id=article_id, + payload=record.payload, + minhash=list(record.minhash), + published_date=record.published_date, + ) + ) + db.flush() + return record.content_hash, True + + +def near_duplicate_groups( + db: Session, + since: datetime | None = None, + threshold: float | None = None, +) -> list[list[str]]: + """ + Group stored observations that differ only in boilerplate. + + ``threshold`` defaults to ``settings.near_duplicate_threshold``, which is + the configured value and the one thing about the grouping that is meant to + be tuned -- see ``src/sources/minhash.py`` on why the signature parameters + are not. + + Returns groups of content hashes, singletons included, in a deterministic + order. + """ + query = db.query(Observation.content_hash, Observation.minhash) + if since is not None: + query = query.filter(Observation.published_date >= since) + signatures = {row[0]: tuple(row[1] or ()) for row in query.all()} + return group_near_duplicates( + signatures, + settings.near_duplicate_threshold if threshold is None else threshold, + ) diff --git a/src/sources/store.py b/src/sources/store.py index bed3cb5..c8aba82 100644 --- a/src/sources/store.py +++ b/src/sources/store.py @@ -14,6 +14,20 @@ tests/sources/test_dedup.py rather than assumed. Added 2026-08-26 for backlog task 005. + +**Amended 2026-08-31 (backlog task 014): this is also where an Observation is +written.** Every adapter run reaches the database here, so putting the +observation write here -- and only here -- is what makes "adapters emit +observations through one path" a property of the code rather than a rule to +remember. The article row and the observation row are written in the same +transaction, linked by ``observations.article_id``. See +``src/database/models.py`` for the standing rule on ``articles`` versus +``observations``. + +An already-stored article still gets its observation checked, not skipped: the +two tables have independent identities (``(feed_id, guid)`` and a content hash), +and a re-run that found the article present but the observation missing should +write the observation rather than assume they agree. """ from __future__ import annotations @@ -26,6 +40,7 @@ from ..database.models import Article, RSSFeed from ..utils import utcnow from .base import RawItem +from .observation import store_observation logger = logging.getLogger(__name__) @@ -56,15 +71,20 @@ def store_items(db: Session, source: RSSFeed, items: Iterable[RawItem]) -> tuple """ Insert items that this source has not already stored. - Returns ``(inserted, duplicates)``. Duplicates are found two ways: an + Returns ``(inserted, duplicates)``, counting *articles* -- the return shape + is unchanged from task 005 and its callers. Duplicates are found two ways: an existing row with the same ``(feed_id, guid)``, and a repeat within this batch (two Federal Register queries can return the same document). The ``_feed_guid_uc`` constraint is left as the backstop -- if it ever fires, something concurrent wrote the same guid and the caller should hear about it as an error rather than have it counted as a duplicate and forgotten. + + Observations written are logged rather than returned, for that reason. The + count is a property of the observations table and is read from it. """ inserted = 0 duplicates = 0 + observations = 0 seen_in_batch: set[str] = set() for item in items: @@ -80,13 +100,21 @@ def store_items(db: Session, source: RSSFeed, items: Iterable[RawItem]) -> tuple existing = ( db.query(Article.id).filter(Article.feed_id == source.id, Article.guid == guid).first() ) - if existing is not None: + if existing is None: + article = Article(feed_id=source.id, **fields) + db.add(article) + db.flush() + article_id = int(article.id) + inserted += 1 + else: + article_id = int(existing[0]) duplicates += 1 - continue - db.add(Article(feed_id=source.id, **fields)) - db.flush() - inserted += 1 + _, created = store_observation(db, source, item, article_id=article_id) + observations += int(created) + + if observations: + logger.info(f"{source.name}: stored {observations} new observation(s)") source.last_fetched = utcnow() # type: ignore[assignment] source.last_error = None # type: ignore[assignment] diff --git a/tests/cli/test_replay_cli.py b/tests/cli/test_replay_cli.py new file mode 100644 index 0000000..f7c1642 --- /dev/null +++ b/tests/cli/test_replay_cli.py @@ -0,0 +1,194 @@ +""" +Tests for the `replay` CLI command (backlog task 014). + +Two of these are the acceptance criteria stated as commands rather than as +functions, because that is how the criteria are written: + +* :class:`TestCommitIsRequired` -- a replay without ``--commit`` writes nothing, + proven by counting rows before and after, not by reading the code. +* :class:`TestOutputIsReproducible` -- the same prompt version twice prints the + same bytes, and a different prompt version prints a visible diff. + +Every adjudicator here is loaded through ``--adjudicator``, which is the +documented seam for a version that is not in the registry. No API key is set or +needed. +""" + +from contextlib import contextmanager +from unittest.mock import patch + +import pytest + +from src.cli.replay import replay_command +from src.database.models import Evidence +from src.evidence import commit, rebuild + +from ..evidence.stubs import KeywordAdjudicator, add_observations, add_watches + +V1 = "tests.evidence.stubs:make_v1" +V2 = "tests.evidence.stubs:make_v2" + + +def _patch_db(session): + @contextmanager + def _ctx(): + try: + yield session + session.flush() + except Exception: + session.rollback() + raise + + return patch("src.cli.replay.get_db", _ctx) + + +@pytest.fixture +def corpus(test_session): + """ + A stored corpus with v1's evidence already committed. + + Built from tests/evidence/stubs.py rather than duplicated, so the CLI tests + and the replay tests cannot drift apart about what the corpus contains. + """ + add_watches(test_session) + add_observations(test_session) + commit(test_session, "v1", rebuild(test_session, KeywordAdjudicator("v1"))) + test_session.flush() + return test_session + + +class TestCommitIsRequired: + def test_a_replay_without_commit_writes_nothing(self, cli_runner, test_session, corpus): + test_session.query(Evidence).delete() + test_session.flush() + assert test_session.query(Evidence).count() == 0 + + with _patch_db(test_session): + result = cli_runner.invoke( + replay_command, ["--prompt-version", "v1", "--adjudicator", V1] + ) + + assert result.exit_code == 0, result.output + assert test_session.query(Evidence).count() == 0 + assert "Nothing was written" in result.output + + def test_the_same_replay_with_commit_writes(self, cli_runner, test_session, corpus): + test_session.query(Evidence).delete() + test_session.flush() + + with _patch_db(test_session): + result = cli_runner.invoke( + replay_command, ["--prompt-version", "v1", "--adjudicator", V1, "--commit"] + ) + + assert result.exit_code == 0, result.output + assert test_session.query(Evidence).count() > 0 + assert "committed v1" in result.output + + def test_a_changed_version_does_not_touch_the_stored_one_without_commit( + self, cli_runner, test_session, corpus + ): + before = [ + (row.observation_hash, row.watch_id, row.direction, row.magnitude) + for row in test_session.query(Evidence).all() + ] + + with _patch_db(test_session): + result = cli_runner.invoke( + replay_command, + ["--prompt-version", "v2", "--against", "v1", "--adjudicator", V2], + ) + + assert result.exit_code == 0, result.output + after = [ + (row.observation_hash, row.watch_id, row.direction, row.magnitude) + for row in test_session.query(Evidence).all() + ] + assert after == before + + +class TestOutputIsReproducible: + def test_the_same_prompt_version_twice_prints_the_same_bytes( + self, cli_runner, test_session, corpus + ): + with _patch_db(test_session): + first = cli_runner.invoke( + replay_command, ["--prompt-version", "v1", "--adjudicator", V1] + ) + second = cli_runner.invoke( + replay_command, ["--prompt-version", "v1", "--adjudicator", V1] + ) + + assert first.exit_code == second.exit_code == 0 + assert first.output == second.output + assert "No difference from the stored evidence." in first.output + + def test_a_changed_prompt_version_prints_a_diff(self, cli_runner, test_session, corpus): + with _patch_db(test_session): + result = cli_runner.invoke( + replay_command, + ["--prompt-version", "v2", "--against", "v1", "--adjudicator", V2], + ) + + assert result.exit_code == 0, result.output + assert "prompt-version=v2 against=v1" in result.output + assert " changed" in result.output + assert "was [supports" in result.output + assert "now [contradicts" in result.output + + def test_the_diff_carries_no_timestamp_or_row_id(self, cli_runner, test_session, corpus): + with _patch_db(test_session): + result = cli_runner.invoke( + replay_command, + ["--prompt-version", "v2", "--against", "v1", "--adjudicator", V2], + ) + assert "created_at" not in result.output + assert "id=" not in result.output + + +class TestRefusals: + def test_an_unregistered_prompt_version_fails_loudly(self, cli_runner, test_session, corpus): + with _patch_db(test_session): + result = cli_runner.invoke(replay_command, ["--prompt-version", "not-a-version"]) + + assert result.exit_code != 0 + assert "no adjudicator is registered" in result.output + + def test_a_mislabelled_adjudicator_is_refused(self, cli_runner, test_session, corpus): + """ + --prompt-version must be the name the adjudicator answers to. + + Otherwise the column that makes evidence reviewable records a name the + thing that produced it never used. + """ + with _patch_db(test_session): + result = cli_runner.invoke( + replay_command, ["--prompt-version", "v9", "--adjudicator", V1] + ) + + assert result.exit_code != 0 + assert "does not answer to" in result.output + + def test_prompt_version_is_required(self, cli_runner): + result = cli_runner.invoke(replay_command, []) + assert result.exit_code != 0 + assert "--prompt-version" in result.output + + def test_an_empty_observation_corpus_says_so(self, cli_runner, test_session): + with _patch_db(test_session): + result = cli_runner.invoke(replay_command, ["--prompt-version", "null-v0"]) + assert result.exit_code == 0 + assert "nothing to replay" in result.output + + +class TestRegisteredInTheApp: + def test_replay_is_a_top_level_command(self): + from src.cli.app import COMMAND_DISPATCH, cli + + assert COMMAND_DISPATCH["replay"] is replay_command + assert "replay" in cli.commands + + def test_commit_is_a_flag_that_defaults_to_off(self): + flag = next(p for p in replay_command.params if p.name == "do_commit") + assert flag.is_flag is True + assert flag.default is False diff --git a/tests/database/test_add_observations_and_evidence.py b/tests/database/test_add_observations_and_evidence.py new file mode 100644 index 0000000..2c928ca --- /dev/null +++ b/tests/database/test_add_observations_and_evidence.py @@ -0,0 +1,163 @@ +""" +Tests for the migration that adds observations and evidence (backlog task 014). + +Weighted, like the watches migration's tests, towards the properties that would +be silently lost if the model-driven creation were ever replaced by hand-written +DDL. For ``observations`` that is not only the CHECK constraints but the BEFORE +UPDATE trigger: a table created without it looks correct in every way except the +one that matters. + +The other thing asserted here is what the migration does *not* do. It must not +touch ``articles``. +""" + +import pytest +from sqlalchemy import create_engine, inspect, text + +from src.database.migrations import add_observations_and_evidence as mig +from src.database.models import Base, Evidence, Observation + + +@pytest.fixture +def blank_engine(tmp_path): + return create_engine(f"sqlite:///{tmp_path / 'fresh.db'}") + + +@pytest.fixture +def corpus_engine(tmp_path): + """A database that already has articles in it, as the real one does.""" + engine = create_engine(f"sqlite:///{tmp_path / 'corpus.db'}") + Base.metadata.create_all(bind=engine) + with engine.begin() as conn: + conn.execute(text("DROP TABLE evidence")) + conn.execute(text("DROP TABLE observations")) + conn.execute( + text("INSERT INTO articles (guid, title) VALUES ('legacy-1', 'A pre-rewrite row')") + ) + return engine + + +class TestUpgrade: + def test_creates_both_tables(self, blank_engine): + assert mig.upgrade(blank_engine) == ["observations", "evidence"] + inspector = inspect(blank_engine) + assert inspector.has_table("observations") + assert inspector.has_table("evidence") + + def test_is_idempotent(self, blank_engine): + mig.upgrade(blank_engine) + assert mig.upgrade(blank_engine) == [] + + def test_columns_match_the_models(self, blank_engine): + mig.upgrade(blank_engine) + inspector = inspect(blank_engine) + assert {c["name"] for c in inspector.get_columns("observations")} == set( + Observation.__table__.columns.keys() + ) + assert {c["name"] for c in inspector.get_columns("evidence")} == set( + Evidence.__table__.columns.keys() + ) + + def test_the_immutability_trigger_is_in_the_created_database(self, blank_engine): + """ + Created from the model, so the trigger rides along. This is the reason + the DDL is not written out in the migration. + """ + mig.upgrade(blank_engine) + with blank_engine.connect() as conn: + triggers = conn.execute( + text( + "SELECT name FROM sqlite_master WHERE type='trigger' AND tbl_name='observations'" + ) + ).fetchall() + assert [row[0] for row in triggers] == ["observations_are_immutable"] + + def test_evidence_records_the_prompt_version_and_refuses_a_blank_one(self, blank_engine): + mig.upgrade(blank_engine) + with blank_engine.connect() as conn: + ddl = conn.execute( + text("SELECT sql FROM sqlite_master WHERE type='table' AND name='evidence'") + ).scalar_one() + + assert "prompt_version" in ddl + assert "ck_evidence_prompt_version_present" in ddl + assert "ck_evidence_direction" in ddl + assert "ck_evidence_magnitude_range" in ddl + assert "_evidence_observation_watch_version_uc" in ddl + + def test_observations_ddl_carries_the_hash_prefix_check(self, blank_engine): + mig.upgrade(blank_engine) + with blank_engine.connect() as conn: + ddl = conn.execute( + text("SELECT sql FROM sqlite_master WHERE type='table' AND name='observations'") + ).scalar_one() + assert "ck_observations_hash_prefix" in ddl + + +class TestArticlesAreUntouched: + def test_the_migration_does_not_change_the_articles_table(self, corpus_engine): + """ + The 55,249-row corpus is not migrated, not rewritten, not dropped. + + Task 014's out-of-scope list says so by name, and the reason the tables + coexist is in src/database/models.py. + """ + with corpus_engine.connect() as conn: + before_ddl = conn.execute( + text("SELECT sql FROM sqlite_master WHERE type='table' AND name='articles'") + ).scalar_one() + before_rows = conn.execute(text("SELECT count(*) FROM articles")).scalar_one() + + mig.upgrade(corpus_engine) + + with corpus_engine.connect() as conn: + after_ddl = conn.execute( + text("SELECT sql FROM sqlite_master WHERE type='table' AND name='articles'") + ).scalar_one() + after_rows = conn.execute(text("SELECT count(*) FROM articles")).scalar_one() + + assert after_ddl == before_ddl + assert after_rows == before_rows == 1 + + def test_the_downgrade_leaves_articles_alone_too(self, corpus_engine): + mig.upgrade(corpus_engine) + mig.downgrade(corpus_engine, confirmed=True) + with corpus_engine.connect() as conn: + assert conn.execute(text("SELECT count(*) FROM articles")).scalar_one() == 1 + + +class TestDowngrade: + def test_refuses_without_confirm(self, blank_engine): + mig.upgrade(blank_engine) + with pytest.raises(SystemExit) as exc: + mig.downgrade(blank_engine) + assert "--confirm" in str(exc.value) + assert inspect(blank_engine).has_table("observations") + + def test_drops_both_with_confirm_in_dependency_order(self, blank_engine): + mig.upgrade(blank_engine) + assert mig.downgrade(blank_engine, confirmed=True) == ["evidence", "observations"] + inspector = inspect(blank_engine) + assert not inspector.has_table("observations") + assert not inspector.has_table("evidence") + + def test_is_idempotent(self, blank_engine): + mig.upgrade(blank_engine) + mig.downgrade(blank_engine, confirmed=True) + assert mig.downgrade(blank_engine, confirmed=True) == [] + + +class TestCommandLine: + def test_up_then_down_with_confirm(self, blank_engine, monkeypatch): + monkeypatch.setattr(mig, "engine", blank_engine) + assert mig.main([]) == 0 + assert inspect(blank_engine).has_table("observations") + assert mig.main(["--down", "--confirm"]) == 0 + assert not inspect(blank_engine).has_table("observations") + + def test_down_without_confirm_exits_with_the_help(self, blank_engine, monkeypatch): + monkeypatch.setattr(mig, "engine", blank_engine) + mig.main([]) + with pytest.raises(SystemExit): + mig.main(["--down"]) + assert inspect(blank_engine).has_table("observations") diff --git a/tests/database/test_connection.py b/tests/database/test_connection.py index a7fbe36..2af0b34 100644 --- a/tests/database/test_connection.py +++ b/tests/database/test_connection.py @@ -116,11 +116,18 @@ def test_create_tables_creates_all_tables(self, tmp_path, mocker): inspector = inspect(engine) table_names = inspector.get_table_names() - # Exactly three, asserted as a set rather than a subset: backlog task - # 012 deleted nineteen models, and a subset check would pass again the - # moment one of them was reintroduced by accident. `watches` was added - # deliberately by task 013 (2026-08-31) and is the only addition since. - assert set(table_names) == {"rss_feeds", "articles", "watches"} + # Asserted as a set rather than a subset: backlog task 012 deleted + # nineteen models, and a subset check would pass again the moment one of + # them was reintroduced by accident. `watches` was added deliberately by + # task 013 (2026-08-31); `observations` and `evidence` by task 014 the + # same day. Every addition here should be a task that says so. + assert set(table_names) == { + "rss_feeds", + "articles", + "watches", + "observations", + "evidence", + } def test_create_tables_is_idempotent(self, tmp_path, mocker): """create_tables should be safe to call multiple times""" diff --git a/tests/database/test_models.py b/tests/database/test_models.py index 15edfd5..ee3420e 100644 --- a/tests/database/test_models.py +++ b/tests/database/test_models.py @@ -190,9 +190,16 @@ def test_only_the_declared_tables_are_mapped(self): ``create_tables()`` would recreate a concept the rewrite deleted, so the mapped set is pinned rather than left to review. - ``watches`` joined the set on 2026-08-31 (backlog task 013). Every - addition to this set should be a task that says so. + ``watches`` joined the set on 2026-08-31 (backlog task 013); + ``observations`` and ``evidence`` on 2026-08-31 (backlog task 014). + Every addition to this set should be a task that says so. """ from src.database.models import Base - assert set(Base.metadata.tables) == {"rss_feeds", "articles", "watches"} + assert set(Base.metadata.tables) == { + "rss_feeds", + "articles", + "watches", + "observations", + "evidence", + } diff --git a/tests/evidence/__init__.py b/tests/evidence/__init__.py new file mode 100644 index 0000000..e69de29 diff --git a/tests/evidence/conftest.py b/tests/evidence/conftest.py new file mode 100644 index 0000000..35d4af7 --- /dev/null +++ b/tests/evidence/conftest.py @@ -0,0 +1,27 @@ +""" +Fixtures for the replay harness, built from :mod:`tests.evidence.stubs`. + +**Nothing here makes an LLM call and nothing here needs an API key.** The whole +point of the harness is that adjudication is pluggable and that the replay +machinery can be exercised without the stochastic part; a suite that needed a +key to test the reproducibility of a replay would be testing the wrong thing. +""" + +import pytest + +from .stubs import add_observations, add_watches + + +@pytest.fixture +def watches(test_session): + return add_watches(test_session) + + +@pytest.fixture +def source(test_session): + return add_observations(test_session) + + +@pytest.fixture +def observations(test_session, source): + return source diff --git a/tests/evidence/stubs.py b/tests/evidence/stubs.py new file mode 100644 index 0000000..907ac83 --- /dev/null +++ b/tests/evidence/stubs.py @@ -0,0 +1,165 @@ +""" +Stub adjudicators and corpus builders for the replay harness. + +**Nothing here makes an LLM call and nothing here needs an API key.** The whole +point of the harness is that adjudication is pluggable and that the replay +machinery can be exercised without the stochastic part; a suite that needed a +key to test the reproducibility of a replay would be testing the wrong thing. +The stubs below are ordinary Python objects with a ``prompt_version`` and an +``adjudicate`` method, which is the entire contract. + +This is a plain module rather than a conftest so that both tests/evidence/ and +tests/cli/test_replay_cli.py can build the same corpus from one definition. +pytest refuses to register one conftest under two names, and a second copy of +the corpus would let the two suites drift. +""" + +from datetime import date, datetime + +from src.database.models import Watch +from src.evidence import Verdict +from src.sources.base import RawItem +from src.sources.store import ensure_source, store_items + +SOURCE_URL = "https://example.gov/api/documents" + + +class KeywordAdjudicator: + """ + A deterministic stand-in: emits a verdict when a watch's terms appear. + + This is a *test* adjudicator and lives in the test tree on purpose. It is + not shipped, because a shipped rule that guessed a direction from a keyword + would write fabricated judgements into the table whose reason for existing + is to make judgements reviewable. What it does have is the property under + test: the same inputs give the same outputs, every time. + """ + + def __init__(self, prompt_version: str, direction: str = "supports", scale: float = 1.0): + self.prompt_version = prompt_version + self.direction = direction + self.scale = scale + + def adjudicate(self, observation, watches): + verdicts = [] + haystack = f"{observation.title} {observation.text}".lower() + for watch in sorted(watches, key=lambda w: w.id): + terms = [ + term.lower() + for clause in (watch.triggers or []) + for term in clause.get("terms", []) + ] + hits = [term for term in terms if term in haystack] + if not hits: + continue + verdicts.append( + Verdict( + watch_id=watch.id, + direction=self.direction, + magnitude=round(min(1.0, self.scale * len(hits) / max(1, len(terms))), 4), + rationale=f"matched {len(hits)} of {len(terms)} term(s)", + ) + ) + return verdicts + + +class CountingAdjudicator: + """Not deterministic: its magnitude drifts with every call. Used to prove + the harness notices, rather than assuming nobody will ever write one.""" + + def __init__(self, prompt_version: str): + self.prompt_version = prompt_version + self.calls = 0 + + def adjudicate(self, observation, watches): # noqa: ARG002 + self.calls += 1 + if not watches: + return [] + return [ + Verdict( + watch_id=sorted(w.id for w in watches)[0], + direction="supports", + magnitude=min(1.0, self.calls / 10), + ) + ] + + +def make_v1() -> KeywordAdjudicator: + """Factory used by the CLI tests via --adjudicator.""" + return KeywordAdjudicator("v1") + + +def make_v2() -> KeywordAdjudicator: + """A 'changed prompt': same matching, opposite direction, half magnitude.""" + return KeywordAdjudicator("v2", direction="contradicts", scale=0.5) + + +WATCH_ROWS = ( + { + "id": "conmon-scope-expands", + "claim": "Continuous monitoring scope expands before the assessment window.", + "belief": 0.35, + "decision_key": "renew-authorization", + "so_what": "Scope expansion moves renewal past the point where lapsing is cheaper.", + "triggers": [{"terms": ["continuous monitoring", "conmon"]}], + "expires": date(2027, 3, 31), + "staleness_alert_days": 30, + }, + { + "id": "hiring-market-tightens", + "claim": "Compliance engineers get harder to hire.", + "belief": 0.5, + "decision_key": "hire-engineer", + "so_what": "A tighter market moves the hiring decision forward.", + "triggers": [{"terms": ["compliance engineer"]}], + "expires": date(2026, 12, 15), + "staleness_alert_days": 45, + }, +) + +CORPUS_ITEMS = ( + RawItem( + guid="doc-1", + url="https://example.gov/1", + title="Agency expands continuous monitoring scope", + normalized_content="The agency expanded continuous monitoring requirements today.", + published_date=datetime(2026, 8, 20, 9, 0), + ), + RawItem( + guid="doc-2", + url="https://example.gov/2", + title="Hiring notice", + normalized_content="The office is seeking a compliance engineer for the program.", + published_date=datetime(2026, 8, 21, 9, 0), + ), + RawItem( + guid="doc-3", + url="https://example.gov/3", + title="Persimmons ripen after the first frost", + normalized_content="Fall fruit guidance for the county extension office.", + published_date=datetime(2026, 8, 22, 9, 0), + ), + RawItem( + guid="doc-4", + url="https://example.gov/4", + title="Notice of meeting", + normalized_content="A routine notice with no bearing on any watch.", + published_date=datetime(2026, 8, 23, 9, 0), + ), +) + + +def add_watches(session): + """Two watches whose trigger terms two of the four corpus items match.""" + rows = [Watch(**fields) for fields in WATCH_ROWS] + session.add_all(rows) + session.flush() + return rows + + +def add_observations(session): + """Four observations, written through the one adapter store path.""" + source = ensure_source(session, "Example Agency", SOURCE_URL, "federal") + store_items(session, source, list(CORPUS_ITEMS)) + session.flush() + return source diff --git a/tests/evidence/test_replay.py b/tests/evidence/test_replay.py new file mode 100644 index 0000000..08ea1b3 --- /dev/null +++ b/tests/evidence/test_replay.py @@ -0,0 +1,290 @@ +""" +The replay proof: deterministic, diffable, and writes nothing without --commit. + +Backlog task 014's central acceptance criterion is that a replay is +reproducible. It is worthless otherwise, and -- the task's own warning -- it +breaks in the direction of looking fine, so the reproducibility is asserted +directly here on the rows and again in tests/cli/test_replay_cli.py on the +printed bytes. +""" + +import pytest + +from src.database.models import Evidence +from src.evidence import ( + NULL_PROMPT_VERSION, + NondeterministicReplay, + NullAdjudicator, + ObservationView, + UnknownPromptVersion, + Verdict, + commit, + diff, + format_diff, + known_prompt_versions, + load_adjudicator, + rebuild, + register, + resolve, + stored_evidence, +) + +from .stubs import CountingAdjudicator, KeywordAdjudicator + + +class TestReplayIsDeterministic: + def test_the_same_prompt_version_twice_produces_identical_rows( + self, test_session, watches, observations + ): + first = rebuild(test_session, KeywordAdjudicator("v1")) + second = rebuild(test_session, KeywordAdjudicator("v1")) + + assert first == second + assert first, "the fixture corpus should produce some evidence" + + def test_a_fresh_adjudicator_instance_gives_the_same_answer( + self, test_session, watches, observations + ): + """Determinism must be a property of the version, not of one object.""" + assert rebuild(test_session, KeywordAdjudicator("v1")) == rebuild( + test_session, KeywordAdjudicator("v1") + ) + + def test_the_diff_of_a_version_against_itself_is_empty( + self, test_session, watches, observations + ): + replayed = rebuild(test_session, KeywordAdjudicator("v1")) + commit(test_session, "v1", replayed) + + again = rebuild(test_session, KeywordAdjudicator("v1")) + assert diff(again, stored_evidence(test_session, "v1")).is_empty + + def test_row_order_is_stable_and_sorted(self, test_session, watches, observations): + rows = rebuild(test_session, KeywordAdjudicator("v1")) + assert rows == sorted(rows) + assert [r.key for r in rows] == sorted(r.key for r in rows) + + def test_the_formatted_diff_is_byte_identical_across_runs( + self, test_session, watches, observations + ): + def render(): + replayed = rebuild(test_session, KeywordAdjudicator("v1")) + result = diff(replayed, stored_evidence(test_session, "v1")) + return format_diff(result, "v1", "v1", 4, 2) + + assert render() == render() + + +class TestAChangedPromptVersionShowsADiff: + def test_a_changed_version_produces_a_visible_diff(self, test_session, watches, observations): + commit(test_session, "v1", rebuild(test_session, KeywordAdjudicator("v1"))) + + v2 = rebuild(test_session, KeywordAdjudicator("v2", direction="contradicts", scale=0.5)) + result = diff(v2, stored_evidence(test_session, "v1")) + + assert not result.is_empty + assert result.changed, "same keys, different verdicts, so these are changes" + assert not result.added and not result.removed + for old, new in result.changed: + assert old.direction == "supports" + assert new.direction == "contradicts" + + def test_a_version_that_finds_nothing_shows_every_row_as_removed( + self, test_session, watches, observations + ): + stored = rebuild(test_session, KeywordAdjudicator("v1")) + commit(test_session, "v1", stored) + + result = diff(rebuild(test_session, NullAdjudicator()), stored_evidence(test_session, "v1")) + assert len(result.removed) == len(stored) + assert not result.added and not result.changed + + def test_the_diff_text_names_what_moved(self, test_session, watches, observations): + commit(test_session, "v1", rebuild(test_session, KeywordAdjudicator("v1"))) + v2 = rebuild(test_session, KeywordAdjudicator("v2", direction="contradicts", scale=0.5)) + text = format_diff(diff(v2, stored_evidence(test_session, "v1")), "v2", "v1", 4, 2) + + assert "prompt-version=v2 against=v1" in text + assert "changed" in text + assert "was [supports" in text + assert "now [contradicts" in text + + +class TestNothingIsWrittenWithoutCommit: + def test_rebuild_writes_no_evidence(self, test_session, watches, observations): + before = test_session.query(Evidence).count() + rows = rebuild(test_session, KeywordAdjudicator("v1")) + + assert rows + assert test_session.query(Evidence).count() == before == 0 + + def test_diffing_writes_no_evidence(self, test_session, watches, observations): + rows = rebuild(test_session, KeywordAdjudicator("v1")) + diff(rows, stored_evidence(test_session, "v1")) + assert test_session.query(Evidence).count() == 0 + + def test_commit_is_what_writes(self, test_session, watches, observations): + rows = rebuild(test_session, KeywordAdjudicator("v1")) + written = commit(test_session, "v1", rows) + + assert written["inserted"] == len(rows) + assert test_session.query(Evidence).count() == len(rows) + + +class TestCommitBehaviour: + def test_every_committed_row_records_the_prompt_version( + self, test_session, watches, observations + ): + commit(test_session, "v1", rebuild(test_session, KeywordAdjudicator("v1"))) + versions = {row.prompt_version for row in test_session.query(Evidence).all()} + assert versions == {"v1"} + + def test_committing_twice_inserts_nothing_the_second_time( + self, test_session, watches, observations + ): + rows = rebuild(test_session, KeywordAdjudicator("v1")) + commit(test_session, "v1", rows) + second = commit(test_session, "v1", rebuild(test_session, KeywordAdjudicator("v1"))) + + assert second == {"inserted": 0, "deleted": 0} + assert test_session.query(Evidence).count() == len(rows) + + def test_committing_a_second_version_leaves_the_first_alone( + self, test_session, watches, observations + ): + v1_rows = rebuild(test_session, KeywordAdjudicator("v1")) + commit(test_session, "v1", v1_rows) + commit( + test_session, + "v2", + rebuild(test_session, KeywordAdjudicator("v2", direction="contradicts", scale=0.5)), + ) + + assert stored_evidence(test_session, "v1") == sorted(v1_rows) + assert len(stored_evidence(test_session, "v2")) == len(v1_rows) + + def test_commit_deletes_rows_the_version_no_longer_produces( + self, test_session, watches, observations + ): + commit(test_session, "v1", rebuild(test_session, KeywordAdjudicator("v1"))) + assert test_session.query(Evidence).count() > 0 + + written = commit(test_session, "v1", []) + assert written["inserted"] == 0 + assert written["deleted"] > 0 + assert test_session.query(Evidence).count() == 0 + + def test_a_nondeterministic_version_is_refused_and_writes_nothing( + self, test_session, watches, observations + ): + drifting = CountingAdjudicator("drifty") + commit(test_session, "drifty", rebuild(test_session, drifting)) + before = test_session.query(Evidence).count() + assert before > 0 + + with pytest.raises(NondeterministicReplay, match="not deterministic"): + commit(test_session, "drifty", rebuild(test_session, drifting)) + + assert test_session.query(Evidence).count() == before + + def test_evidence_rows_are_rebuilt_from_observations_alone( + self, test_session, watches, observations + ): + """ + Every evidence row points at a stored observation, by hash. + + Nothing in the replay path reads the articles table; this asserts the + result, and the observation hashes are the only join key that exists. + """ + from src.database.models import Observation + + commit(test_session, "v1", rebuild(test_session, KeywordAdjudicator("v1"))) + hashes = {row[0] for row in test_session.query(Observation.content_hash).all()} + for row in test_session.query(Evidence).all(): + assert row.observation_hash in hashes + + +class TestRebuildScope: + def test_limit_narrows_the_corpus(self, test_session, watches, observations): + everything = rebuild(test_session, KeywordAdjudicator("v1")) + assert len(rebuild(test_session, KeywordAdjudicator("v1"), limit=1)) <= len(everything) + + def test_an_empty_watch_set_produces_no_evidence(self, test_session, observations): + assert rebuild(test_session, KeywordAdjudicator("v1")) == [] + + def test_the_adjudicator_sees_only_the_stored_payload( + self, test_session, watches, observations + ): + seen = [] + + class Recorder: + prompt_version = "recorder" + + def adjudicate(self, observation, watches): # noqa: ARG002 + seen.append(observation) + return [] + + rebuild(test_session, Recorder()) + assert seen + assert all(isinstance(view, ObservationView) for view in seen) + assert all(set(view.payload) and view.content_hash.startswith("sha256:") for view in seen) + + +class TestVerdictValidation: + def test_direction_must_be_in_the_vocabulary(self): + with pytest.raises(ValueError, match="direction must be one of"): + Verdict(watch_id="w", direction="maybe", magnitude=0.5) + + def test_magnitude_is_bounded(self): + with pytest.raises(ValueError, match="magnitude must be in"): + Verdict(watch_id="w", direction="supports", magnitude=1.5) + + def test_a_verdict_must_name_a_watch(self): + with pytest.raises(ValueError, match="must name the watch"): + Verdict(watch_id=" ", direction="supports", magnitude=0.5) + + +class TestAdjudicatorRegistry: + def test_the_null_adjudicator_is_registered_and_finds_nothing(self): + adjudicator = resolve(NULL_PROMPT_VERSION) + assert adjudicator.prompt_version == NULL_PROMPT_VERSION + assert adjudicator.adjudicate(ObservationView("sha256:x", {}), []) == [] + + def test_an_unregistered_version_fails_loudly_and_names_what_is_known(self): + with pytest.raises(UnknownPromptVersion, match="no adjudicator is registered"): + resolve("does-not-exist") + + def test_rebinding_a_version_is_refused(self): + with pytest.raises(ValueError, match="already registered"): + register(NULL_PROMPT_VERSION, NullAdjudicator) + + def test_the_registry_ships_no_llm_adjudicator(self): + """ + The adjudication prompt is backlog task 016, not this task. + + Asserted so that a shipped placeholder cannot quietly become the thing + producing evidence. + """ + assert known_prompt_versions() == [NULL_PROMPT_VERSION] + + def test_an_adjudicator_can_be_loaded_by_path(self): + adjudicator = load_adjudicator("tests.evidence.stubs:make_v1") + assert adjudicator.prompt_version == "v1" + + def test_a_bad_path_is_refused(self): + with pytest.raises(ValueError, match="module:attr"): + load_adjudicator("no_colon_here") + + def test_a_path_to_a_non_adjudicator_is_refused(self): + with pytest.raises(TypeError, match="did not produce an Adjudicator"): + load_adjudicator("tests.evidence.stubs:SOURCE_URL") + + def test_replay_needs_no_api_key(self, test_session, watches, observations, monkeypatch): + """ + The harness must run with no ANTHROPIC_API_KEY set. + + Out of scope for this task by name: "Any LLM call: replay must work with + no ANTHROPIC_API_KEY, driven by a stub adjudicator in tests." + """ + monkeypatch.delenv("ANTHROPIC_API_KEY", raising=False) + assert rebuild(test_session, KeywordAdjudicator("v1")) diff --git a/tests/sources/fixtures/near_duplicates.json b/tests/sources/fixtures/near_duplicates.json new file mode 100644 index 0000000..dfb0a78 --- /dev/null +++ b/tests/sources/fixtures/near_duplicates.json @@ -0,0 +1,38 @@ +{ + "near_duplicate_a": { + "article_id": 32416, + "feed": "Prince William Living", + "guid": "https://princewilliamliving.com/?p=142458", + "url": "https://princewilliamliving.com/love-takes-center-stage-at-the-8th-annual-valentines-day-bash-at-the-manassas-museum-2/?utm_source=rss&utm_medium=rss&utm_campaign=love-takes-center-stage-at-the-8th-annual-valentines-day-bash-at-the-manassas-museum-2", + "title": "Love Takes Center Stage at the 8th Annual Valentine’s Day Bash at the Manassas Museum", + "published_date": "2026-01-22 16:36:18.000000", + "normalized_content": "Provided by PWC Circuit Court Clerk’s Office | Photo Credit: Mark Summerson // Mark Summerson Photography – @marksummersonphoto IG On Feb. 14, 2026, hearts will be joined in love as Prince William County Circuit Court Clerk Jacqueline Smith hosts the 8th annual Valentine’s Day Bash at the Manassas Museum. This year’s celebration is anticipated to be the region’s most romantic and captivating Valentine’s Day event of the year, offering couples an elegant and affordable alternative to a traditional wedding. The Valentine’s Day Bash reimagines courtroom-style civil ceremonies with a touch of sophistication, transforming them into unforgettable experiences. More than a nod to Cupid’s arrow, the event celebrates lasting commitments and cherished memories, bringing the community together in a shared moment of happiness. The Manassas Museum’s Mae Merchant Hall will be transformed into a romantic haven, complete with whimsical décor and heart-shaped details. Couples will exchange vows throughout the day in intimate civil ceremonies designed to feel timeless and personal. Every couple’s story adds to the romance of the day. Some pen their own vows, others renew their love after decades of marriage, and others, like former school sweethearts, begin new chapters together. Each ceremony unfolds as a treasured moment, woven into the rich tapestry of love celebrated. “Our Annual Valentine’s Day Bash offers couples a special, elegant, and affordable way to celebrate their union together with family and friends in a welcoming and supportive environment” Jacqueline Smith said. “Our community partners and small businesses come together year after year for this event to make sure all of our couples enjoy every aspect of their very special day.” To schedule your marriage license application and ceremony at the 8th Annual Valentine’s Day Bash, please contact the Circuit Court Clerk’s Office at 703-792-6036 or email CircuitCourt@pwcgov.org. Businesses interested in participating in the event, please reach out to Candi Choi at cchoi@pwcgov.org. The post Love Takes Center Stage at the 8th Annual Valentine’s Day Bash at the Manassas Museum appeared first on Prince William Living." + }, + "near_duplicate_b": { + "article_id": 33941, + "feed": "What's Up Prince William (Woodbridge)", + "guid": "https://whatsupwoodbridge.com/?p=28479", + "url": "https://whatsupwoodbridge.com/valentines-day-bash-manassas-museum/", + "title": "Love Takes Center Stage at the 8th Annual Valentine’s Day Bash at the Manassas Museum", + "published_date": "2026-01-23 15:20:25.000000", + "normalized_content": "MANASSAS, Va. — On February 14, 2026, hearts will be joined in love as Prince William County Circuit Court Clerk Jacqueline Smith hosts the 8th annual Valentine’s Day Bash at the Manassas Museum. This year’s celebration is anticipated to be the region’s most romantic and captivating Valentine’s Day event of the year, offering couples an elegant and affordable alternative to a traditional wedding. The Valentine’s Day Bash reimagines courtroom-style civil ceremonies with a touch of sophistication, transforming them into unforgettable experiences. More than a nod to Cupid’s arrow, the event celebrates lasting commitments and cherished memories, bringing the community together in a shared moment of happiness. The Manassas Museum’s Mae Merchant Hall will be transformed into a romantic haven, complete with whimsical décor and heart-shaped details. Couples will exchange vows throughout the day in intimate civil ceremonies designed to feel timeless and personal. Every couple’s story adds to the romance of the day. Some pen their own vows, others renew their love after decades of marriage, and others, like former school sweethearts, begin new chapters together. Each ceremony unfolds as a treasured moment, woven into the rich tapestry of love celebrated. “Our Annual Valentine’s Day Bash offers couples a special, elegant, and affordable way to celebrate their union together with family and friends in a welcoming and supportive environment” Jacqueline Smith said. “Our community partners and small businesses come together year after year for this event to make sure all of our couples enjoy every aspect of their very special day.” To schedule your marriage license application and ceremony at the 8th Annual Valentine’s Day Bash, please contact the Circuit Court Clerk’s Office at 703-792-6036 or email CircuitCourt@pwcgov.org. Businesses interested in participating in the event, please reach out to Candi Choi at cchoi@pwcgov.org. The post Love Takes Center Stage at the 8th Annual Valentine’s Day Bash at the Manassas Museum appeared first on What's Up Prince William." + }, + "distinct_a": { + "article_id": 21083, + "feed": "Prince William Living", + "guid": "https://princewilliamliving.com/?p=141748", + "url": "https://princewilliamliving.com/sealed-with-love-couples-celebrate-their-vows-at-the-circuit-court-clerks-winter-wedding-celebration/?utm_source=rss&utm_medium=rss&utm_campaign=sealed-with-love-couples-celebrate-their-vows-at-the-circuit-court-clerks-winter-wedding-celebration", + "title": "Sealed with Love: Couples Celebrate their Vows at the Circuit Court Clerk’s Winter Wedding Celebration", + "published_date": "2025-12-30 16:22:18.000000", + "normalized_content": "Provided by Prince William County Circuit Court Clerk’s Office Couples let their love shine this winter amid the charm and warmth of the Town of Occoquan. On Friday, Dec. 12, 2025, Prince William County Circuit Court Clerk Jacqueline C. Smith, in collaboration with the Town of Occoquan, hosted the second annual Seal the Deal, a signature civil wedding experience. Set against Occoquan’s historic streets and waterfront views, the event took place at Madigan’s Waterfront, a fine dining restaurant along the Occoquan River. Nineteen couples exchanged vows in courthouse-style ceremonies beneath a handcrafted arch, surrounded by family and friends. The elegant simplicity of the ceremonies kept the focus on each couple, offering a romantic alternative to traditional wedding planning. Following the ceremonies, couples enjoyed celebratory refreshments and “wedding cupcakes” at Madigan’s. Local businesses contributed to the festivities, transforming Seal the Deal into a full community celebration. The Circuit Court Clerk’s Office extends gratitude to the Town of Occoquan and the local business community for their generosity in providing wedding gifts to the couples. Contributions came from Madigan’s Waterfront, Ballywhack Shack, Brandon’s Flowers, Curate, Evidi Massage, Forever Young, Glory Be, Hello Jackson, Hitchcock Paper, Jerry’s Occoquan Jewelers, Leaf & Petal, Mark Summerson Photography, and Occoquan Outfitter. Together, these partners helped create a day where love stories unfolded in a town sparkling with romance, community, and holiday magic. In February 2026, the Clerk will host her annual Valentine’s Day Bash, inviting couples to come together once again for romantic civil unions. For updates and information about civil ceremony events, follow @PWCCircuitCourt on social media or visit pwcva.gov/department/circuit-court. The post Sealed with Love: Couples Celebrate their Vows at the Circuit Court Clerk’s Winter Wedding Celebration appeared first on Prince William Living." + }, + "distinct_b": { + "article_id": 16723, + "feed": "Prince William Living", + "guid": "https://princewilliamliving.com/?p=140682", + "url": "https://princewilliamliving.com/fall-focus-on-fruit-persimmons/?utm_source=rss&utm_medium=rss&utm_campaign=fall-focus-on-fruit-persimmons", + "title": "Fall Focus on Fruit: Persimmons", + "published_date": "2025-11-17 19:57:26.000000", + "normalized_content": "Provided by Cakes by Happy Eatery Persimmons are orange fruits that are in season in the fall and winter featuring multiple varieties that originated in China centuries ago.  There are two main varieties: the fuyu persimmon looks like an orange squat tomato or a miniature pumpkin, and the hachiya persimmon is larger and elongated (kind of like a big, orange acorn) or heart shaped. Both varieties are sweet when ripe and taste like a cross between a papaya and a peach.  Emily Wu-Rorrer of Cakes By Happy Eatery went on Fox 5’s Good Day DC earlier today to demonstrate some of the best ways to take advantage of this cool weather treat. One important difference between the two varieties: fresh fuyu is non-astringent and can be eaten firm or ripe, almost like an apple with a peach/mango flavor.  Hachiya is very astringent and must be eaten ripe, otherwise Wu-Rorrer says “your mouth will experience a dry, puckering sensation and even feel a little numb!”.  Once ripe, persimmons to not keep well, so enjoy quickly! “We love baking with them,” says Wu-Rorrer.  “You want to puree the ripe fruit with the skin on, and blend into a pulp.  I like to do bread or muffins with them.”  The bakery also likes to do an upside-down cake with them similar to the pineapple version or broil them halved and drizzled with honey.  Persimmon pie is also an option and presents similar to pumpkin. Once you’ve exhausted your baking options and still have some left, why not do a smoothie?  Wu-Rorrer says slice of the stem, cut into wedges (keep the skin if ripe), and then she likes to go with a milk alternative like almond, oat or soy milk with some cinnamon or nutmeg. For more information about persimmons and other seasonal options, click here.  You can also follow the bakery on Instagram or Facebook. The post Fall Focus on Fruit: Persimmons appeared first on Prince William Living." + } +} diff --git a/tests/sources/test_minhash.py b/tests/sources/test_minhash.py new file mode 100644 index 0000000..b542494 --- /dev/null +++ b/tests/sources/test_minhash.py @@ -0,0 +1,172 @@ +""" +Near-duplicate detection, checked against four real articles from the corpus. + +The four items in ``fixtures/near_duplicates.json`` were pulled from the +operator's 55,249-row database on 2026-08-31, not written for this test: + +* ``near_duplicate_a`` / ``near_duplicate_b`` -- the same press release about a + county clerk's Valentine's Day event, carried by two different feeds (Prince + William Living and What's Up Prince William). The bodies are identical; they + differ in the leading photo credit and in the "appeared first on " + footer. Exactly the case content-addressing cannot catch, because the bytes + genuinely differ. +* ``distinct_a`` / ``distinct_b`` -- two articles from the *same* publisher. + ``distinct_a`` is deliberately the hard case: it is about the same county + clerk, the same kind of civil wedding event, carries the same publisher + boilerplate, and even mentions the Valentine's Day Bash by name. If a + threshold is going to over-group anything, it will over-group this. + +Fabricated near-duplicates would have proved the algorithm runs. These prove it +separates the two cases on text nobody wrote to be separable. +""" + +import json +from pathlib import Path + +import pytest + +from src.config.settings import settings +from src.sources.minhash import ( + PERMUTATIONS, + group_near_duplicates, + shingles, + signature, + similarity, +) + +FIXTURES = Path(__file__).parent / "fixtures" + + +@pytest.fixture(scope="module") +def corpus() -> dict: + return json.loads((FIXTURES / "near_duplicates.json").read_text(encoding="utf-8")) + + +@pytest.fixture(scope="module") +def signatures(corpus) -> dict: + return { + key: signature(f"{item['title']} {item['normalized_content']}") + for key, item in corpus.items() + } + + +class TestRealNearDuplicates: + def test_the_two_near_duplicates_score_far_above_the_threshold(self, signatures): + score = similarity(signatures["near_duplicate_a"], signatures["near_duplicate_b"]) + assert score > settings.near_duplicate_threshold + # Measured 0.891 on 2026-08-31. Asserted loosely so that a change to the + # shingle size shows up as a failure here rather than as a silent drift. + assert score == pytest.approx(0.89, abs=0.05) + + def test_the_two_distinct_items_score_far_below_the_threshold(self, signatures): + score = similarity(signatures["distinct_a"], signatures["distinct_b"]) + assert score < settings.near_duplicate_threshold + assert score == pytest.approx(0.0, abs=0.05) + + def test_the_hardest_distinct_pair_still_does_not_group(self, signatures): + """Same publisher, same subject, same boilerplate, different article.""" + score = similarity(signatures["near_duplicate_a"], signatures["distinct_a"]) + assert score < settings.near_duplicate_threshold + + def test_grouping_puts_the_pair_together_and_leaves_the_others_alone(self, signatures): + groups = group_near_duplicates(signatures, settings.near_duplicate_threshold) + assert groups == [ + ["distinct_a"], + ["distinct_b"], + ["near_duplicate_a", "near_duplicate_b"], + ] + + def test_the_configured_threshold_sits_in_the_gap(self, signatures): + """ + The threshold is only meaningful if there is room on both sides of it. + + Measured: the near-duplicate pair is 0.89, the closest distinct pair is + 0.016. Any threshold in that gap gives the same grouping, which is what + makes 0.7 a choice rather than a fitted constant. + """ + near = similarity(signatures["near_duplicate_a"], signatures["near_duplicate_b"]) + worst_distinct = max( + similarity(signatures[a], signatures[b]) + for a, b in ( + ("near_duplicate_a", "distinct_a"), + ("near_duplicate_a", "distinct_b"), + ("near_duplicate_b", "distinct_a"), + ("near_duplicate_b", "distinct_b"), + ("distinct_a", "distinct_b"), + ) + ) + assert worst_distinct < settings.near_duplicate_threshold < near + assert near - worst_distinct > 0.5 + + +class TestSignatureProperties: + def test_a_signature_is_the_declared_width(self, signatures): + assert all(len(sig) == PERMUTATIONS for sig in signatures.values()) + + def test_the_same_text_always_signs_the_same(self, corpus): + text = corpus["distinct_b"]["normalized_content"] + assert signature(text) == signature(text) + + def test_a_document_is_identical_to_itself(self, signatures): + assert similarity(signatures["distinct_a"], signatures["distinct_a"]) == 1.0 + + def test_empty_text_has_an_empty_signature_and_matches_nothing(self, signatures): + assert signature("") == () + assert signature(" ") == () + assert similarity((), ()) == 0.0 + assert similarity((), signatures["distinct_a"]) == 0.0 + + def test_short_text_still_produces_one_shingle(self): + assert len(shingles("three short words")) == 1 + assert len(signature("three short words")) == PERMUTATIONS + + def test_comparing_signatures_of_different_widths_raises(self): + with pytest.raises(ValueError, match="different PERMUTATIONS"): + similarity((1, 2, 3), (1, 2, 3, 4)) + + +class TestGrouping: + def test_singletons_are_returned_too(self): + sigs = {"a": signature("wholly unrelated text about persimmons and frost dates")} + assert group_near_duplicates(sigs, 0.7) == [["a"]] + + def test_the_output_order_does_not_depend_on_input_order(self, signatures): + forward = group_near_duplicates(signatures, 0.7) + reversed_input = dict(reversed(list(signatures.items()))) + assert group_near_duplicates(reversed_input, 0.7) == forward + + def test_grouping_is_transitive(self): + base = "the agency finalized the continuous monitoring rule this morning " * 10 + sigs = { + "a": signature(base), + "b": signature(base + " Filed by the desk."), + "c": signature("Filed by the desk. " + base), + } + assert group_near_duplicates(sigs, 0.7) == [["a", "b", "c"]] + + def test_an_out_of_range_threshold_is_refused(self, signatures): + with pytest.raises(ValueError, match="must be in"): + group_near_duplicates(signatures, 1.5) + + def test_an_empty_corpus_groups_to_nothing(self): + assert group_near_duplicates({}, 0.7) == [] + + +class TestConfiguredThreshold: + def test_the_threshold_is_configuration_not_a_literal(self): + assert 0.0 < settings.near_duplicate_threshold < 1.0 + assert settings.near_duplicate_threshold == 0.7 + + def test_the_threshold_is_overridable_by_environment(self, monkeypatch): + from src.config.settings import Settings + + monkeypatch.setenv("NEAR_DUPLICATE_THRESHOLD", "0.95") + assert Settings().near_duplicate_threshold == 0.95 + + def test_a_stricter_threshold_splits_the_near_duplicate_pair(self, signatures): + assert group_near_duplicates(signatures, 0.95) == [ + ["distinct_a"], + ["distinct_b"], + ["near_duplicate_a"], + ["near_duplicate_b"], + ] diff --git a/tests/sources/test_observation.py b/tests/sources/test_observation.py new file mode 100644 index 0000000..a3b261b --- /dev/null +++ b/tests/sources/test_observation.py @@ -0,0 +1,328 @@ +""" +The Observation invariants: content-addressing, immutability, one write path. + +The first class here is the one the task named as the landmine to test first. +An adapter that lets a per-fetch value into the hashed payload stores the same +document on every run, and the symptom is "dedup looks broken" rather than +"hashing is broken", so it is checked directly and against the clock. +""" + +from dataclasses import replace +from datetime import datetime, timedelta + +import pytest +from sqlalchemy import text + +from src.database.models import Article, Observation, ObservationIsImmutable, RSSFeed +from src.sources.base import ARTICLE_FIELDS, RawItem +from src.sources.observation import ( + OBSERVATION_FIELDS, + observation_hash, + observation_payload, + observe, + store_observation, +) +from src.sources.store import ensure_source, store_items +from src.utils import utcnow + +SOURCE_URL = "https://example.gov/api/documents" + + +BASE_ITEM = RawItem( + guid="doc-2026-0001", + url="https://example.gov/documents/2026-0001", + title="Agency finalizes the continuous monitoring rule", + description="A summary of the final rule.", + content="

The agency finalized the rule today.

", + normalized_content="The agency finalized the rule today.", + published_date=datetime(2026, 8, 20, 9, 30), + author="Office of the Federal Register", + categories=("rules",), + language="en", +) + + +def make_item(**overrides) -> RawItem: + """One adapter item, with any field replaced. RawItem is frozen, so this + is a replace() rather than a kwargs splat -- which also keeps the field + names checkable.""" + return replace(BASE_ITEM, **overrides) + + +@pytest.fixture +def source(test_session): + return ensure_source(test_session, "Example Agency", SOURCE_URL, "federal") + + +class TestHashIsContentOnly: + """ + The hash must be a function of the content and of nothing else. + + Landmine, backlog task 014: "an adapter that includes a fetch timestamp, + session ID, or any per-fetch value in the hashed payload will store the same + article repeatedly, and the dedup will look broken when the hashing is what + is wrong. Test that first." + """ + + def test_the_same_item_hashes_the_same_at_two_different_times(self): + item = make_item() + first = observe(SOURCE_URL, item) + second = observe(SOURCE_URL, item) + assert first.content_hash == second.content_hash + + def test_re_fetching_hours_later_does_not_change_the_hash(self, test_session, source): + """ + Store, move the wall clock forward, store again. Same hash, one row. + + observed_at is what changes between the two writes, and observed_at is + deliberately not in the payload. + """ + item = make_item() + first_hash, created = store_observation(test_session, source, item) + assert created is True + + row = test_session.query(Observation).one() + # Rewriting observed_at directly is refused, so a second store is used + # to prove the hash does not move; the point is that the second call + # computes the same identity for the same content. + second_hash, created_again = store_observation(test_session, source, item) + + assert second_hash == first_hash + assert created_again is False + assert test_session.query(Observation).count() == 1 + assert row.observed_at is not None + + def test_no_per_fetch_column_is_in_the_payload(self): + """observed_at and the article clock columns must not be payload keys.""" + payload = observation_payload(SOURCE_URL, make_item()) + for banned in ("observed_at", "fetched_at", "created_at", "updated_at", "last_fetched"): + assert banned not in payload + + def test_payload_keys_are_exactly_the_declared_field_set(self): + payload = observation_payload(SOURCE_URL, make_item()) + assert set(payload) == set(OBSERVATION_FIELDS) + + def test_the_field_set_extends_the_task_005_seam(self): + """ + OBSERVATION_FIELDS is ARTICLE_FIELDS minus word_count plus source_url. + + Asserted rather than described, so that a field added to the RSS + normalizer (which tests/sources/test_base.py pins ARTICLE_FIELDS to) + cannot end up stored-but-unhashed. + """ + assert set(OBSERVATION_FIELDS) == (set(ARTICLE_FIELDS) - {"word_count"}) | {"source_url"} + + def test_changing_any_content_field_changes_the_hash(self): + base = observe(SOURCE_URL, make_item()).content_hash + assert observe(SOURCE_URL, make_item(title="Something else")).content_hash != base + assert observe(SOURCE_URL, make_item(normalized_content="Other body")).content_hash != base + assert ( + observe(SOURCE_URL, make_item(published_date=datetime(2026, 8, 21))).content_hash + != base + ) + + def test_the_same_text_from_two_sources_is_two_observations(self): + """ + Who published it is part of what was observed. + + Grouping the pair back together is the MinHash signature's job, not the + hash's -- see tests/sources/test_minhash.py. + """ + item = make_item() + assert ( + observe(SOURCE_URL, item).content_hash != observe("https://other/", item).content_hash + ) + + def test_a_payload_with_an_unhashed_key_is_refused(self): + payload = observation_payload(SOURCE_URL, make_item()) + payload["session_id"] = "abc123" + with pytest.raises(ValueError, match="every stored key must be hashed"): + observation_hash(payload) + + def test_the_payload_is_json_round_trippable(self): + import json + + payload = observation_payload(SOURCE_URL, make_item()) + assert json.loads(json.dumps(payload)) == payload + + +class TestImmutability: + def test_the_orm_refuses_to_update_a_stored_observation(self, test_session, source): + store_observation(test_session, source, make_item()) + test_session.commit() + + row = test_session.query(Observation).one() + row.payload = {"tampered": True} + + with pytest.raises(ObservationIsImmutable): + test_session.flush() + + def test_raw_sql_update_is_refused_by_the_trigger(self, test_session, source): + """ + The guarantee has to hold for writes that never touch the ORM. + + A trigger is what makes this true for the sqlite3 shell, a hand-written + migration, and anything else that issues an UPDATE. + """ + stored_hash, _ = store_observation(test_session, source, make_item()) + test_session.commit() + + with pytest.raises(Exception, match="observations are immutable"): + test_session.execute( + text("UPDATE observations SET payload = :p WHERE content_hash = :h"), + {"p": '{"tampered": true}', "h": stored_hash}, + ) + test_session.rollback() + + def test_the_row_is_unchanged_after_a_refused_update(self, test_session, source): + stored_hash, _ = store_observation(test_session, source, make_item()) + test_session.commit() + before = dict(test_session.query(Observation).one().payload) + + with pytest.raises(Exception, match="observations are immutable"): + test_session.execute( + text("UPDATE observations SET payload = :p WHERE content_hash = :h"), + {"p": '{"tampered": true}', "h": stored_hash}, + ) + test_session.rollback() + + test_session.expire_all() + assert dict(test_session.query(Observation).one().payload) == before + + def test_storing_the_same_content_again_is_a_no_op_not_an_update(self, test_session, source): + store_observation(test_session, source, make_item()) + test_session.commit() + _, created = store_observation(test_session, source, make_item()) + assert created is False + assert test_session.query(Observation).count() == 1 + + +class TestOneWritePath: + def test_only_the_observation_module_constructs_an_observation(self): + """ + Adapters emit observations through one path, checked against the tree. + + The guarantee that observations are content-addressed is only as strong + as the number of places that can write one, so the count is asserted + rather than trusted. src/database/models.py is excluded because that is + where the class is defined. + """ + from pathlib import Path + + root = Path(__file__).resolve().parents[2] / "src" + allowed = {root / "sources" / "observation.py", root / "database" / "models.py"} + + offenders = [ + str(path.relative_to(root)) + for path in sorted(root.rglob("*.py")) + if path not in allowed and "Observation(" in path.read_text(encoding="utf-8") + ] + assert offenders == [], ( + f"these modules construct an Observation directly: {offenders}. " + f"Every write goes through src/sources/observation.py::store_observation." + ) + + def test_the_adapter_store_path_writes_an_observation_per_article(self, test_session, source): + items = [make_item(), make_item(guid="doc-2", title="A different rule", url="u2")] + inserted, duplicates = store_items(test_session, source, items) + + assert (inserted, duplicates) == (2, 0) + assert test_session.query(Observation).count() == 2 + + def test_each_observation_links_to_the_article_written_beside_it(self, test_session, source): + store_items(test_session, source, [make_item()]) + + observation = test_session.query(Observation).one() + article = test_session.query(Article).one() + assert observation.article_id == article.id + assert observation.source_id == source.id + + def test_a_second_adapter_run_inserts_neither_an_article_nor_an_observation( + self, test_session, source + ): + store_items(test_session, source, [make_item()]) + inserted, duplicates = store_items(test_session, source, [make_item()]) + + assert (inserted, duplicates) == (0, 1) + assert test_session.query(Article).count() == 1 + assert test_session.query(Observation).count() == 1 + + def test_an_article_that_predates_observations_gains_one_on_the_next_run( + self, test_session, source + ): + """ + The two tables have independent identities, so a re-run repairs the gap. + + This is the path a legacy article takes if the same item is fetched + again: the article is a duplicate, but the observation is written. + """ + item = make_item() + test_session.add(Article(feed_id=source.id, **item.as_article_fields())) + test_session.flush() + assert test_session.query(Observation).count() == 0 + + inserted, duplicates = store_items(test_session, source, [item]) + + assert (inserted, duplicates) == (0, 1) + assert test_session.query(Observation).count() == 1 + + def test_the_published_date_projection_matches_the_payload(self, test_session, source): + store_items(test_session, source, [make_item()]) + row = test_session.query(Observation).one() + assert row.published_date.isoformat() == row.payload["published_date"] + + +class TestNearDuplicateQuery: + def test_groups_are_read_from_the_stored_signatures(self, test_session, source): + from src.sources.observation import near_duplicate_groups + + body = "The agency finalized the continuous monitoring rule today " * 12 + store_items( + test_session, + source, + [ + make_item(guid="a", url="a", normalized_content=body), + make_item(guid="b", url="b", normalized_content=body + " Filed by the desk."), + make_item( + guid="c", + url="c", + title="Unrelated", + normalized_content="Persimmons ripen in the fall and are best after frost. " + * 8, + ), + ], + ) + + groups = near_duplicate_groups(test_session) + sizes = sorted(len(group) for group in groups) + assert sizes == [1, 2] + + def test_the_window_filter_narrows_the_corpus(self, test_session, source): + from src.sources.observation import near_duplicate_groups + + old = make_item(guid="old", url="old", published_date=datetime(2026, 1, 1)) + new = make_item(guid="new", url="new", published_date=datetime(2026, 8, 20)) + store_items(test_session, source, [old, new]) + + recent = near_duplicate_groups(test_session, since=datetime(2026, 6, 1)) + assert sum(len(group) for group in recent) == 1 + + +class TestSourceRowUnchanged: + def test_storing_still_stamps_the_source(self, test_session, source): + before = utcnow() - timedelta(seconds=1) + store_items(test_session, source, [make_item()]) + assert source.last_fetched >= before + assert source.error_count == 0 + + +class TestFeedIsolation: + def test_two_feeds_carrying_the_same_item_store_two_observations(self, test_session): + first = ensure_source(test_session, "Feed One", "https://one.example/rss", "news") + second = ensure_source(test_session, "Feed Two", "https://two.example/rss", "news") + + store_items(test_session, first, [make_item()]) + store_items(test_session, second, [make_item()]) + + assert test_session.query(Observation).count() == 2 + assert test_session.query(RSSFeed).count() == 2 From 1a21e6e7f71fe4a61584fa607361c68f07cd9f3d Mon Sep 17 00:00:00 2001 From: sam-aydlette Date: Mon, 31 Aug 2026 12:32:45 -0400 Subject: [PATCH 2/2] backlog: queue closing the legacy article write path Task 014's reviewer confirmed the one-write-path invariant holds as shipped and flagged the residual: RSSFetcher.fetch_and_store_feed writes articles directly with no observation, and fetch_all_active_feeds still reaches it. It has no in-repo caller since task 012 deleted the orchestrator, which is precisely what makes it dangerous -- it is live, exported, and the obvious thing to reach for when refreshing the corpus. I ran it twice on 2026-08-28. Queued as 025 and marked BLOCKS 015, because routing reads observations: a corpus refreshed through the legacy path produces articles that can never route, and the failure is silent -- the coverage gap would look like a source problem rather than an ingestion one. --- backlog/025-close-the-legacy-write-path.md | 26 ++++++++++++++++++++++ 1 file changed, 26 insertions(+) create mode 100644 backlog/025-close-the-legacy-write-path.md diff --git a/backlog/025-close-the-legacy-write-path.md b/backlog/025-close-the-legacy-write-path.md new file mode 100644 index 0000000..7340f9e --- /dev/null +++ b/backlog/025-close-the-legacy-write-path.md @@ -0,0 +1,26 @@ +# Close the legacy article write path that bypasses observations, before Tier 1 depends on the corpus being complete. +REPO: InsightWeaver +STATUS: QUEUED # QUEUED | IN_PROGRESS | PARKED | DONE | FAILED +SIZE: small +BLOCKS: 015-tier1-deterministic-routing +ACCEPTANCE: `make check` passes, plus: `RSSFetcher.fetch_and_store_feed` either routes through `src/sources/store.py:store_items` so every article it writes gets an observation in the same transaction, **or** raises rather than writing -- whichever the implementer judges correct, with the reasoning in the PR; the "one write path" test from task 014 is extended so it would fail if a second article-writing path is reintroduced; and a test drives the legacy entry point and asserts the observation count moves in step with the article count, or that the call refuses. +OUT OF SCOPE: Backfilling observations for the 55,249 pre-existing articles -- that is a separate, bounded, mechanical task and task 014 states the rule that makes it safe to defer. Rewriting the RSS fetcher, changing its normalization, or touching `src/sources/rss_adapter.py`, which already goes through the correct path. Deleting `src/rss/` wholesale. +LANDMINES: **This is unreachable today and that is exactly why it is dangerous.** Task 012 deleted the orchestrator that called `fetch_all_active_feeds`, so no in-repo caller remains and the reviewer confirmed it -- but the function is live, exported, and is the obvious thing to reach for when someone wants to refresh the corpus. The lead ran that path twice on 2026-08-28 to fetch feeds. An article written through it has **no observation**, so Tier 1 cannot route it, and the failure is silent: the corpus grows, routing finds nothing, and the coverage gap looks like a source problem rather than an ingestion one. **Do not fix this by adding a second observation write.** Task 014's invariant is that exactly one code path constructs an `Observation`; a parallel write here satisfies the symptom and destroys the property. `main` is protected -- open a PR. +--- +Written 2026-08-31, from task 014's review. + +Task 014 established that adapters emit observations through exactly one path, enforced by a test +that greps the source tree. The reviewer confirmed the property holds **as shipped**, and flagged +the one residual: `src/rss/fetcher.py`'s `fetch_and_store_feed` writes articles directly, and +`fetch_all_active_feeds` still reaches it. + +The IC disclosed this rather than widening its own diff to fix it, which was correct. It is closed +here because task 015 makes it consequential: routing reads observations, so a corpus refreshed +through the legacy path produces articles that can never route, and nothing surfaces that. + +The likely right answer is to raise. The RSS adapter already covers this ground and goes through +`store_items`; a second, older entry point that does the same job worse is a hazard whether or not +it currently has a caller. But the implementer should check whether anything outside this +repository -- a cron entry, a systemd timer, a shell script on the operator's machine -- calls it +before removing the ability to call it at all. The reviewer could only verify there is no in-repo +caller.