Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
9 changes: 8 additions & 1 deletion Makefile
Original file line number Diff line number Diff line change
@@ -1,4 +1,4 @@
.PHONY: help install install-dev test lint format fmt typecheck check pre-commit clean clean-all coverage db-add-watches db-drop-briefing update-deps
.PHONY: help install install-dev test lint format fmt typecheck check pre-commit clean clean-all coverage db-add-watches db-add-observations db-drop-briefing update-deps

# Tool resolution (added 2026-08-24).
# Before this, every recipe called bare `pytest` / `ruff` / `mypy`, which only
Expand Down Expand Up @@ -45,6 +45,7 @@ help:
@echo ""
@echo "Database:"
@echo " make db-add-watches Create the watches table (additive, safe)"
@echo " make db-add-observations Create observations and evidence (additive, safe)"
@echo " make db-drop-briefing Drop the deleted briefing product's tables"
@echo " (DESTRUCTIVE; captures to a dump first)"
@echo ""
Expand Down Expand Up @@ -120,6 +121,12 @@ clean-all: clean
db-add-watches:
$(PYTHON) -m src.database.migrations.add_watches_table

# Additive in the same way, and specifically does NOT touch the 55,249-row
# articles table: observations coexist with it under the rule written in
# src/database/models.py. (2026-08-31, backlog task 014.)
db-add-observations:
$(PYTHON) -m src.database.migrations.add_observations_and_evidence

# Deliberately does NOT pass --confirm. The migration refuses without it, so
# running this target prints what it would destroy and stops; the operator types
# the real command themselves. A Makefile target that drops tables on `make` is
Expand Down
62 changes: 62 additions & 0 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -522,6 +522,68 @@ beat can select it.

---

## Observations, Evidence, and replay

Added 2026-08-31 (`backlog/014`). Adjudication -- deciding whether an item bears
on a pre-registered Watch -- is the only stochastic component in the system.
Everything either side of it is deterministic and unit-testable. This layer is
what makes the stochastic part reviewable: hold the inputs fixed, change the
prompt, and diff the outputs.

**Observations are immutable and content-addressed.** An observation is one
thing a source published, keyed by a SHA-256 hash of the normalized adapter
output. Nothing per-fetch is in the hashed payload -- not the fetch time, not a
session id -- so re-fetching an unchanged document is a no-op rather than a
second row. Once written, a row cannot be changed: an ORM update raises
`ObservationIsImmutable`, and a `BEFORE UPDATE` trigger refuses an `UPDATE`
issued from anywhere else, including the `sqlite3` shell.

**Evidence is derived, and every row records the prompt version that produced
it.** `evidence` links `(observation, watch)` with a direction and a magnitude.
The `prompt_version` column is per row, not per run, so "where did this
judgement come from" is answerable six weeks later.

```bash
# Rebuild v2's evidence and diff it against what v1 stored. Writes nothing.
insightweaver replay --prompt-version v2 --against v1

# Re-run v1 against its own stored rows. An empty diff is the reproducibility check.
insightweaver replay --prompt-version v1

# Persist. --commit is required; a replay that wrote by default would destroy
# the before-state at the moment you wanted to compare against it.
insightweaver replay --prompt-version v2 --commit
```

Replay needs no `ANTHROPIC_API_KEY`. The adjudicator is a plug-in seam
(`src/evidence/adjudicator.py`); the adjudication prompt itself is a later task,
and the only version this build ships is `null-v0`, which returns no verdicts.
An adjudicator that guessed a direction from a keyword match would put
fabricated judgements into the table whose whole purpose is to make judgements
reviewable.

**Near-duplicate detection.** Two feeds carrying the same wire story with
different boilerplate have different content hashes, correctly. A MinHash
signature over 5-word shingles is written beside each observation and groups
them: `settings.near_duplicate_threshold` (default 0.7, `NEAR_DUPLICATE_THRESHOLD`)
is the tunable. Measured on two real corpus pairs, a genuine near-duplicate
scores 0.89 and the hardest distinct pair 0.016.

**`observations` versus the 55,249-row `articles` table.** They coexist, with one
stated rule: `articles` is the pre-rewrite archive and keeps the row shape the
older code reads; `observations` is authoritative for everything built from task
014 onward. Exactly one code path writes an observation
(`src/sources/observation.py`, called from the adapter store path), and each new
article gets an observation in the same transaction, linked by
`observations.article_id`. The pre-existing rows have no observation and are not
migrated -- the hash is a pure function of columns they already carry, so a
backfill is mechanical and is left to its own task.

Create the tables with `make db-add-observations` (additive; it does not touch
`articles`).

---

## Requirements

- Python 3.10+
Expand Down
26 changes: 26 additions & 0 deletions backlog/025-close-the-legacy-write-path.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,26 @@
# Close the legacy article write path that bypasses observations, before Tier 1 depends on the corpus being complete.
REPO: InsightWeaver
STATUS: QUEUED # QUEUED | IN_PROGRESS | PARKED | DONE | FAILED
SIZE: small
BLOCKS: 015-tier1-deterministic-routing
ACCEPTANCE: `make check` passes, plus: `RSSFetcher.fetch_and_store_feed` either routes through `src/sources/store.py:store_items` so every article it writes gets an observation in the same transaction, **or** raises rather than writing -- whichever the implementer judges correct, with the reasoning in the PR; the "one write path" test from task 014 is extended so it would fail if a second article-writing path is reintroduced; and a test drives the legacy entry point and asserts the observation count moves in step with the article count, or that the call refuses.
OUT OF SCOPE: Backfilling observations for the 55,249 pre-existing articles -- that is a separate, bounded, mechanical task and task 014 states the rule that makes it safe to defer. Rewriting the RSS fetcher, changing its normalization, or touching `src/sources/rss_adapter.py`, which already goes through the correct path. Deleting `src/rss/` wholesale.
LANDMINES: **This is unreachable today and that is exactly why it is dangerous.** Task 012 deleted the orchestrator that called `fetch_all_active_feeds`, so no in-repo caller remains and the reviewer confirmed it -- but the function is live, exported, and is the obvious thing to reach for when someone wants to refresh the corpus. The lead ran that path twice on 2026-08-28 to fetch feeds. An article written through it has **no observation**, so Tier 1 cannot route it, and the failure is silent: the corpus grows, routing finds nothing, and the coverage gap looks like a source problem rather than an ingestion one. **Do not fix this by adding a second observation write.** Task 014's invariant is that exactly one code path constructs an `Observation`; a parallel write here satisfies the symptom and destroys the property. `main` is protected -- open a PR.
---
Written 2026-08-31, from task 014's review.

Task 014 established that adapters emit observations through exactly one path, enforced by a test
that greps the source tree. The reviewer confirmed the property holds **as shipped**, and flagged
the one residual: `src/rss/fetcher.py`'s `fetch_and_store_feed` writes articles directly, and
`fetch_all_active_feeds` still reaches it.

The IC disclosed this rather than widening its own diff to fix it, which was correct. It is closed
here because task 015 makes it consequential: routing reads observations, so a corpus refreshed
through the legacy path produces articles that can never route, and nothing surfaces that.

The likely right answer is to raise. The RSS adapter already covers this ground and goes through
`store_items`; a second, older entry point that does the same job worse is a hazard whether or not
it currently has a caller. But the implementer should check whether anything outside this
repository -- a cron entry, a systemd timer, a shell script on the operator's machine -- calls it
before removing the ability to call it at all. The reviewer could only verify there is no in-repo
caller.
16 changes: 15 additions & 1 deletion src/cli/app.py
Original file line number Diff line number Diff line change
Expand Up @@ -19,6 +19,7 @@

from .colors import accent, header, muted
from .output import set_debug_mode
from .replay import replay_command
from .sources import sources_command
from .watch import watch_command

Expand All @@ -31,12 +32,16 @@
COMMAND_DISPATCH = {
"sources": sources_command,
"watch": watch_command,
"replay": replay_command,
}


def print_command_refresher():
"""Print a short refresher of available commands."""
click.echo(f"\n{header('Commands:')} {accent('sources')} | {accent('watch')} | help | exit\n")
click.echo(
f"\n{header('Commands:')} {accent('sources')} | {accent('watch')} | "
f"{accent('replay')} | help | exit\n"
)


def print_help():
Expand All @@ -46,6 +51,9 @@ def print_help():
click.echo(
f" {accent('watch')} - Pre-registered watches and the decisions they serve"
)
click.echo(
f" {accent('replay')} - Rebuild Evidence from Observations and diff it"
)
click.echo(f" {accent('help')} - Show this help message")
click.echo(f" {accent('exit')} - Exit InsightWeaver")
click.echo()
Expand All @@ -64,6 +72,12 @@ def print_help():
muted(" Watches are only ever created from those files -- there is no 'watch add'.")
)
click.echo()
click.echo(header("Replay command:"))
click.echo(f" {accent('replay --prompt-version X')} - Rebuild Evidence for prompt version X")
click.echo(
muted(" Prints the diff against stored Evidence and writes nothing. --commit persists.")
)
click.echo()
click.echo(muted("Tip: Add --debug to any command to see detailed logs"))
click.echo()

Expand Down
142 changes: 142 additions & 0 deletions src/cli/replay.py
Original file line number Diff line number Diff line change
@@ -0,0 +1,142 @@
"""
Replay command - rebuild Evidence from Observations and show what changed.

``--commit`` is required to persist, and that is the whole shape of this
command. A replay that wrote by default would destroy the before-state at
exactly the moment you wanted to compare against it: you would have run the
thing that answers "what did my prompt change do" and, in answering, made the
question unanswerable. Printing the diff and requiring one more flag costs a
keystroke and preserves the property the harness exists to provide.

Three ways to use it:

replay --prompt-version v1
Recompute v1 and diff against v1's stored rows. An empty diff is the
reproducibility check: a deterministic adjudicator changes nothing.

replay --prompt-version v2 --against v1
Diff v2's output against v1's stored rows. This is the prompt-change
review.

replay --prompt-version v2 --commit
Make the stored rows for v2 equal what v2 just produced. Touches no other
version's rows, and refuses if v2 disagrees with itself.

Added 2026-08-31 for backlog task 014.
"""

import click

from ..database.connection import get_db
from ..database.models import Observation, Watch
from ..evidence import (
NondeterministicReplay,
UnknownPromptVersion,
commit,
diff,
format_diff,
load_adjudicator,
rebuild,
resolve,
stored_evidence,
)
from .colors import accent, header, muted, warning


def _adjudicator_for(prompt_version: str, path: str | None):
"""
The adjudicator to run, and a check that it is the one being labelled.

A ``--adjudicator`` whose own ``prompt_version`` disagrees with
``--prompt-version`` is refused rather than reconciled. The label written on
every evidence row has to be the name the adjudicator answers to, or the
column is decoration.
"""
try:
adjudicator = load_adjudicator(path) if path else resolve(prompt_version)
except UnknownPromptVersion as exc:
raise click.ClickException(str(exc))
except (ImportError, TypeError, ValueError) as exc:
raise click.ClickException(f"could not load adjudicator {path!r}: {exc}")

if adjudicator.prompt_version != prompt_version:
raise click.ClickException(
f"the adjudicator loaded from {path!r} calls itself "
f"{adjudicator.prompt_version!r}, but --prompt-version says "
f"{prompt_version!r}. Refusing to label its output with a name it does "
f"not answer to."
)
return adjudicator


@click.command(name="replay")
@click.option(
"--prompt-version",
required=True,
help="The adjudication prompt version to run and to label its output with.",
)
@click.option(
"--against",
default=None,
help="Diff against this version's stored evidence instead of --prompt-version's.",
)
@click.option(
"--adjudicator",
"adjudicator_path",
default=None,
help="Load 'module:attr' instead of the registered adjudicator for this version.",
)
@click.option("--limit", type=int, default=None, help="Replay only the first N observations.")
@click.option(
"--commit",
"do_commit",
is_flag=True,
default=False,
help="Persist the replayed evidence. Without this, nothing is written.",
)
def replay_command(prompt_version, against, adjudicator_path, limit, do_commit):
"""Rebuild Evidence from Observations and print the diff. Writes only with --commit."""
baseline_version = against or prompt_version
adjudicator = _adjudicator_for(prompt_version, adjudicator_path)

with get_db() as session:
observations = session.query(Observation).count()
watches = session.query(Watch).count()

if observations == 0:
click.echo(
warning(
"No observations are stored, so there is nothing to replay. "
"Evidence is rebuilt from observations alone; the 55,249 legacy "
"'articles' rows are not observations. See src/database/models.py."
)
)
return

replayed = rebuild(session, adjudicator, limit=limit)
baseline = stored_evidence(session, baseline_version)
result = diff(replayed, baseline)

click.echo(header("REPLAY"))
click.echo(format_diff(result, prompt_version, baseline_version, observations, watches))
click.echo()

if not do_commit:
click.echo(
muted(
"Nothing was written. Re-run with "
+ accent("--commit")
+ muted(" to persist this version's evidence.")
)
)
return

try:
written = commit(session, prompt_version, replayed)
except NondeterministicReplay as exc:
raise click.ClickException(str(exc))

click.echo(
f"committed {prompt_version}: {written['inserted']} inserted, "
f"{written['deleted']} deleted"
)
16 changes: 16 additions & 0 deletions src/config/settings.py
Original file line number Diff line number Diff line change
Expand Up @@ -20,6 +20,22 @@ class Settings(BaseSettings):
os.getenv("SMART_RSS_FETCH_THRESHOLD_MINUTES", "60")
)

# How similar two observations' MinHash signatures must be before they are
# treated as the same item published twice. 0.7 was chosen against measured
# values rather than picked: on the two real corpus pairs in
# tests/sources/fixtures/near_duplicates.json, a genuine near-duplicate (the
# same press release carried by two Prince William feeds, differing only in
# byline and footer) scores 0.89, and the hardest distinct pair -- two
# different articles from the same publisher about the same county clerk's
# wedding events -- scores 0.016. 0.7 sits in the middle of that gap with
# room on both sides for text this suite has not seen.
#
# This is the *only* near-duplicate parameter that belongs in config. The
# signature width and shingle size are format, not preference; changing them
# silently invalidates every signature already stored, which is why they are
# constants in src/sources/minhash.py. (2026-08-31, backlog task 014.)
near_duplicate_threshold: float = float(os.getenv("NEAR_DUPLICATE_THRESHOLD", "0.7"))

project_root: Path = Path(__file__).parent.parent.parent
data_dir: Path = project_root / "data"
logs_dir: Path = project_root / "src" / "logs"
Expand Down
Loading
Loading