From e206456e592d97d40bb715ca933558cce83bf39b Mon Sep 17 00:00:00 2001 From: sam-aydlette Date: Thu, 27 Aug 2026 13:44:52 -0400 Subject: [PATCH 1/4] Add the operator calibration loop: questions add, predict, resolve, forecast --due The ledger held 33 predictions and had graded none. 25 were phrased "X would signal Y" -- interpretation rules, which cannot be wrong -- and only 3 carried a date, so 19 expired unjudged. The ledger was not empty for want of a resolver; it was empty because nothing gradeable had been written into it, and because every command was read-only: the model wrote the graph and the human read it. Four write commands, all local (no API key, no network call): questions add "..." --cadence 90d predict 23 "Yes -- slips" --by 2026-12-31 --confidence 0.7 forecast --due resolve 41 --outcome no --note "..." --by and --confidence are required and rejected at entry, not defaulted. A claim with no resolution date can never come due, so it can never be graded; a confidence with a default is a non-commitment in a different costume. Both refusals store nothing. A cadence is not a deadline. questions.cadence says how often a question is worth re-examining; predictions.due_by says when a specific claim resolves, so a question reviewed quarterly can hold a claim resolving in three weeks. forecast --due tests each question against its own interval, and stamps last_reviewed_at whether or not anything moved -- a quiet question that reappears daily trains the operator to skim. predictions.author (operator | model) splits track-record: a plain hit rate for the operator, model predictions under their own heading, never mixed. Nothing auto-resolves from coverage; the human resolves, and resolved_at is recorded separately from due_by so grading three months late stays visible. A resolved prediction is not editable. Schema changes are in a reversible migration, add_calibration_loop. The predictions half is a SQLite table rebuild (relaxing made_in_synthesis_id to nullable, which ALTER TABLE cannot do) with row counts asserted on both sides; verified carrying all 33 existing rows, backfilled to author='model'. The downgrade refuses rather than misattributing an operator claim. track-record's output changed shape, so its two existing assertions were updated to read the model heading. Interactive dispatch now splits arguments with shlex, without which a multi-word claim reached Click as four arguments. --- README.md | 47 ++ docs/CONCEPTS.md | 24 +- src/cli/app.py | 48 +- src/cli/forecast.py | 157 ++++- src/cli/predictions.py | 140 ++++- src/cli/questions.py | 74 +++ src/cli/stake.py | 238 +++++++ .../migrations/add_calibration_loop.py | 261 ++++++++ src/database/models.py | 61 +- src/utils/cadence.py | 117 ++++ tests/cli/test_calibration_loop.py | 584 ++++++++++++++++++ tests/cli/test_ledger_scope.py | 7 +- tests/cli/test_predictions_cli.py | 13 +- tests/database/test_calibration_migration.py | 173 ++++++ tests/utils/test_cadence.py | 88 +++ 15 files changed, 1988 insertions(+), 44 deletions(-) create mode 100644 src/cli/stake.py create mode 100644 src/database/migrations/add_calibration_loop.py create mode 100644 src/utils/cadence.py create mode 100644 tests/cli/test_calibration_loop.py create mode 100644 tests/database/test_calibration_migration.py create mode 100644 tests/utils/test_cadence.py diff --git a/README.md b/README.md index 60b04b3..17a7c4c 100644 --- a/README.md +++ b/README.md @@ -157,6 +157,53 @@ that decides what a brief looks like. --- +## The calibration loop: staking and grading your own calls + +Every other ledger command reads. These four write, and they are what makes the graph +a calibration instrument rather than a reading list -- the thing being calibrated is +*your* judgement. + +```bash +questions add "Does CMMC Phase 2 slip past its statutory date?" --cadence 90d +predict 23 "Yes -- slips" --by 2026-12-31 --confidence 0.7 +forecast --due +resolve 41 --outcome no --note "DFARS class deviation published 2026-11-14" +``` + +All four are local database work: no `ANTHROPIC_API_KEY`, no network call. The loop is +`forecast --due` (what is due right now, at each question's own speed), grade what +resolved, read the diff on whatever came up for review, stake anything new. **The brief +is optional to that loop** -- ingestion is the fragile half, and the ledger should keep +working on a week where the sources returned nothing worth reading. + +**A date and a confidence are required, and rejected at entry.** Nothing is stored +without them. The tool's first 33 predictions were graded zero times: 25 were phrased +"X would signal Y" -- an interpretation rule, which cannot be wrong -- and only 3 +carried a date, so 19 aged out unjudged. A claim with no resolution date can never come +due. There is no confidence default either; an unstated confidence is a non-commitment +in a different costume. + +**A cadence is not a deadline.** A question's `--cadence` says how often it is worth +re-examining; a prediction's `--by` says when a specific claim resolves. A question +reviewed quarterly can hold a claim resolving in three weeks. `forecast --due` tests +each question against its *own* interval -- a 7d question and a 90d question come due +independently -- and **stamps a question as reviewed whether or not anything moved**, +because a quiet question that reappears every day trains you to skim. + +**`predictions track-record` counts your predictions only.** The model's are reported +under a separate heading and never folded into the figure; a hit rate that blends the +two measures nothing. Nothing auto-resolves from coverage: a tool that grades its +operator's calls using the corpus that produced them is measuring agreement with itself. + +Schema: `questions.cadence`, `questions.last_reviewed_at`, and +`predictions.author | due_by | confidence | outcome`, added by +`python -m src.database.migrations.add_calibration_loop` (reversible; the downgrade +refuses rather than misattributing an operator claim). Commands live in +`src/cli/stake.py`, `src/cli/questions.py` and `src/cli/forecast.py`; interval +arithmetic in `src/utils/cadence.py`. + +--- + ## Beats: briefing a subject instead of a person The user profile models *you* -- a location, a profession, a voting context. A **beat** diff --git a/docs/CONCEPTS.md b/docs/CONCEPTS.md index f9b29ee..32efb55 100644 --- a/docs/CONCEPTS.md +++ b/docs/CONCEPTS.md @@ -18,7 +18,13 @@ A sixth concept, the **Beat**, is not a commitment but a *scope* over them: it s A **Question** is an unresolved epistemic thread the coverage is implicitly tracking. Most situations end with one — "Will the Fed cut rates in June?" — and that question persists across daily runs until it resolves. -A Question has one of two origins. Most are **emergent**: the graph notices what coverage leaves unresolved. A beat can also **declare** one up front — see "Standing questions" below. Both are the same kind of row, and everything downstream (predictions, decision evidence, appearance counts) treats them identically. +A Question has one of three origins. Most are **emergent**: the graph notices what coverage leaves unresolved. A beat can also **declare** one up front — see "Standing questions" below. Since 2026-08-27 the operator can declare one directly with `questions add "..." --cadence 90d`. All three are the same kind of row, and everything downstream (predictions, decision evidence, appearance counts) treats them identically. + +**Cadence.** A question the operator declared carries a `cadence` — a review interval such as `7d`, `90d`, `1y`. **A cadence is not a deadline.** It says how often the question is worth re-examining; a Prediction's `due_by` says when a specific claim resolves. A question reviewed quarterly can hold a claim resolving in three weeks, and that nesting is the point: a CISA directive deadline is a 30-day loop and whether CMMC Phase 2 slips is a multi-year one, and forcing both onto one rhythm checks the fast one too late and buries the slow one in noise. + +`forecast --due` surfaces a question when *its own* interval has elapsed since `last_reviewed_at` (or `first_asked_at`, if never reviewed), and **stamps it whether or not anything moved** — a quiet question that reappears every day trains the operator to skim. A cadence is itself falsifiable: a question always quiet at its interval was set too fast, one that moved twice before its next review was set too slow. + +Cadence is nullable, and null means "not on a review schedule". Emergent questions have none — the interval is the operator's read on how fast a subject moves, so nothing infers it from coverage volume. **When created.** Each situation's `unresolved_questions.primary` (and optional secondaries) is matched against the open question graph by a Haiku call. If today's question is the same underlying question as one already open, today's situation gets bound to that existing Question. Otherwise a new Question is created. @@ -26,7 +32,7 @@ A Question has one of two origins. Most are **emergent**: the graph notices what **Reappearance.** When a previously resolved Question's text appears again in fresh coverage, a *new* Question is created with `previous_question_id` pointing at the resolved one. Resolution is never silently undone. -**CLI.** `questions list [--beat NAME] | show | resolve --note ...` +**CLI.** `questions add "..." --cadence 90d | list [--beat NAME] | show | resolve --note ...`. `questions list` shows each question's cadence and time until next review. **In the brief.** Returning questions surface their identity inline: `Q47 (run 4, asked 2026-03-12)` for repeat appearances, `Q47 (new)` for first ones. @@ -34,15 +40,19 @@ A Question has one of two origins. Most are **emergent**: the graph notices what ## Predictions -A **Prediction** is a falsifiable observable the synthesis committed to watching for. Each `what_to_watch` entry in a situation's output becomes a Prediction keyed to that situation's primary Question. +A **Prediction** is a falsifiable claim about the future. Every prediction carries an `author`: `model` or `operator`. + +**Model predictions.** Each `what_to_watch` entry in a situation's output becomes a Prediction keyed to that situation's primary Question, created during synthesis after questions are resolved. Before each new synthesis, a check pass grades the open ledger against today's coverage: triggered, contradicted, expired (no signal after 90 days), or still open. + +**Operator predictions.** Staked by hand: `predict "..." --by YYYY-MM-DD --confidence 0.7`. Both flags are **required and rejected at entry** — nothing is stored without them. That is not fussiness. Of the first 33 model predictions, 25 were phrased "X would signal Y" (an interpretation rule, which cannot be wrong) and only 3 carried a date; 19 expired unjudged. A claim with no resolution date can never come due, so it can never be graded, and a confidence with a default is a non-commitment in a different costume. -**When created.** During synthesis, after questions are resolved. Each `{observable, trigger_condition}` pair in `where_this_goes.what_to_watch` becomes a Prediction row. +**When an operator prediction resolves.** Manually: `resolve --outcome yes|no --note "..."`. `resolved_at` is recorded **separately from `due_by`** — resolving three months late is itself calibration data. A resolved prediction is not editable, and resolving one twice is refused. -**When resolved.** Before each new synthesis, a check pass grades the open ledger against today's coverage: triggered (observable appeared), contradicted (coverage explicitly went the other way), expired (no signal after 90 days), or still open. +**Auto-resolution from coverage is deliberately absent.** A tool that grades its operator's calls using the same corpus that produced them is measuring agreement with itself. The friction of resolving by hand is the instrument, not an obstacle to it. -**Why it matters.** This makes the tool's forward-looking statements auditable. The `predictions track-record` command shows the calibration: of N predictions resolved in the last 90 days, what fraction triggered vs. were contradicted. +**The author split.** `predictions track-record` reports the operator's plain hit rate — how often the claim, as written, came true — and reports model predictions under a separate heading. The two are never mixed: a track record blending them measures nothing. The model's stay in the ledger as *prompts*, suggestions about what is worth holding an opinion on. -**CLI.** `predictions open | triggered | contradicted | track-record`, each taking `--beat NAME` +**CLI.** `predictions open | triggered | contradicted | track-record`, each taking `--beat NAME`; plus the write side, `predict` and `resolve`. **In the brief.** A transparency line at the top of each brief reports the check results: "Prediction check: 8 open observables graded — 1 triggered, 0 contradicted, 7 still open." diff --git a/src/cli/app.py b/src/cli/app.py index 9376a86..dad6945 100644 --- a/src/cli/app.py +++ b/src/cli/app.py @@ -4,6 +4,7 @@ """ import contextlib +import shlex import time import click @@ -18,15 +19,20 @@ from .predictions import predictions_command from .questions import questions_command from .sources import sources_command +from .stake import predict_command, resolve_command # Maps command prefix to its Click command object. # Order matters: longer prefixes checked first via startswith(). +# In particular "predictions" must precede "predict", since the shorter name is +# a prefix of the longer one (2026-08-27, backlog task 011). COMMAND_DISPATCH = { "brief": brief_group, "forecast": forecast_command, "frames": frames_command, "questions": questions_command, "predictions": predictions_command, + "predict": predict_command, + "resolve": resolve_command, "decisions": decisions_command, "diet": diet_command, "sources": sources_command, @@ -38,6 +44,7 @@ def print_command_refresher(): refresher = ( f"\n{header('Commands:')} {accent('brief')} | {accent('forecast')} | " f"{accent('frames')} | {accent('questions')} | {accent('predictions')} | " + f"{accent('predict')} | {accent('resolve')} | " f"{accent('decisions')} | {accent('diet')} | {accent('sources')} | " f"help | exit\n" ) @@ -54,6 +61,8 @@ def print_help(): click.echo(f" {accent('frames')} - Manage narrative frame glossary") click.echo(f" {accent('questions')} - Inspect the persistent question graph") click.echo(f" {accent('predictions')} - Inspect the predictions ledger") + click.echo(f" {accent('predict')} - Stake your own claim on a question") + click.echo(f" {accent('resolve')} - Record the outcome of a prediction") click.echo(f" {accent('decisions')} - Manage the decision journal") click.echo(f" {accent('diet')} - Inspect your information diet's structure") click.echo(f" {accent('sources')} - Per-source structural calibration") @@ -79,6 +88,8 @@ def print_help(): click.echo() click.echo(header("Forecast command options:")) click.echo(f" {accent('--days N')} - Resolved-record window (default: 60)") + click.echo(f" {accent('--due')} - What is due right now, at each question's") + click.echo(muted(" own cadence. Stamps what it surfaces.")) click.echo() click.echo(header("Frames command:")) click.echo(f" {accent('frames list')} - List all topic clusters and frames") @@ -87,8 +98,9 @@ def print_help(): click.echo(f" {accent('frames gaps')} - Show recurring perspective gaps") click.echo() click.echo(header("Questions command:")) - click.echo(f" {accent('questions list')} - List open (or resolved) questions") + click.echo(f" {accent('questions list')} - List questions, cadence, next review") click.echo(muted(" Add --beat NAME to any ledger view to read that beat's ledger")) + click.echo(f" {accent('questions add')} '...' --cadence 90d - Declare a question you carry") click.echo(f" {accent('questions show')} - Show a question's full history") click.echo(f" {accent('questions resolve')} --note '...' - Resolve a question") click.echo() @@ -96,7 +108,13 @@ def print_help(): click.echo(f" {accent('predictions open')} - Predictions still waiting on coverage") click.echo(f" {accent('predictions triggered')} - Predictions later coverage confirmed") click.echo(f" {accent('predictions contradicted')} - Predictions later coverage went against") - click.echo(f" {accent('predictions track-record')} - Rolling calibration record") + click.echo(f" {accent('predictions track-record')} - Your hit rate; model shown separately") + click.echo() + click.echo(header("Stake and resolve your own claims:")) + click.echo(f" {accent('predict')} '...' --by YYYY-MM-DD --confidence 0.7") + click.echo(muted(" Both flags required; a claim with no date is refused, not stored")) + click.echo(f" {accent('resolve')} --outcome yes|no --note '...'") + click.echo(muted(" You grade your own calls. Nothing grades them for you")) click.echo() click.echo(header("Decisions command:")) click.echo(f" {accent('decisions list')} - List standing decisions") @@ -123,6 +141,10 @@ def print_help(): click.echo(muted(" brief --beat us-public-sector-compliance (subject brief)")) click.echo(muted(" forecast (open observables + last 60d resolved)")) click.echo(muted(" forecast --days 30 (tighter resolved-record window)")) + click.echo(muted(" forecast --due (what is due now, at each question's cadence)")) + click.echo(muted(" questions add 'Does CMMC Phase 2 slip?' --cadence 90d")) + click.echo(muted(" predict 23 'Yes -- slips' --by 2026-12-31 --confidence 0.7")) + click.echo(muted(" resolve 41 --outcome no --note 'class deviation published'")) click.echo(muted(" frames list (view narrative frame glossary)")) click.echo(muted(" frames gaps (view perspective gaps in your feeds)")) click.echo() @@ -135,11 +157,31 @@ def _dispatch_command(command: str) -> bool: Dispatch a user command to the appropriate Click command. Returns True if a command was matched, False otherwise. + + Argument splitting is quote-aware (shlex) as of 2026-08-27: `predict 23 + "Yes -- slips" --by ...` takes a multi-word claim as one argument, and a + plain whitespace split would have handed Click four arguments instead of + one. The same applies to every option that takes prose, such as + `resolve --note`. """ for prefix, click_cmd in COMMAND_DISPATCH.items(): if command.startswith(prefix): + try: + args = shlex.split(command)[1:] + except ValueError as exc: + from .colors import error as err_style + + click.echo(err_style(f"Could not read that command: {exc}")) + print_command_refresher() + return True with contextlib.suppress(SystemExit): - click_cmd.main(command.split()[1:], standalone_mode=False) + try: + click_cmd.main(args, standalone_mode=False) + except click.ClickException as exc: + # standalone_mode=False re-raises instead of printing. + # The write commands reject bad input this way, so the + # rejection has to reach the operator intact. + exc.show() print_command_refresher() return True return False diff --git a/src/cli/forecast.py b/src/cli/forecast.py index 3518fd5..bcccd7a 100644 --- a/src/cli/forecast.py +++ b/src/cli/forecast.py @@ -5,22 +5,25 @@ by whether each one has resolved. """ -from datetime import timedelta +from datetime import datetime, timedelta import click -from ..context.beat_scope import prediction_scope_filter +from ..context.beat_scope import prediction_scope_filter, question_scope_filter from ..database.connection import get_db from ..database.models import ( PREDICTION_STATUS_CONTRADICTED, PREDICTION_STATUS_OPEN, PREDICTION_STATUS_TRIGGERED, + QUESTION_STATUS_OPEN, Prediction, Question, ) from ..utils import utcnow +from ..utils.cadence import InvalidCadence, describe_next_review, is_due from .colors import accent, header, muted, success, warning from .scope import beat_option, resolve_beat_scope, scope_label +from .stake import due_predictions_header, render_due_prediction @click.command(name="forecast") @@ -31,14 +34,29 @@ default=60, help="Window for the resolved track record (default: 60 days).", ) +@click.option( + "--due", + "due_only", + is_flag=True, + default=False, + help=( + "Show only what is due right now: your claims past their resolution " + "date, and questions whose own review interval has elapsed. Surfacing " + "a question stamps it as reviewed." + ), +) @beat_option -def forecast_command(days, beat_name): +def forecast_command(days, due_only, beat_name): """ Show open observables and the recently-resolved record. Scoped: this is a derived view over the predictions ledger, so it inherits the ledger's scoping. Without --beat it shows your own observables only. """ + if due_only: + _render_due(beat_name) + return + cutoff = utcnow() - timedelta(days=days) with get_db() as session: @@ -106,6 +124,139 @@ def forecast_command(days, beat_name): ) +def _render_due(beat_name: str | None) -> None: + """ + Answer "what is due right now", at each question's own speed. + + Two things can be due, and they are deliberately different clocks: + + * a **prediction** whose ``due_by`` has passed -- a specific claim that can + now be graded, and + * a **question** whose own review interval has elapsed since it was last + reviewed -- worth re-examining, whether or not anything moved. + + Surfacing a question stamps ``last_reviewed_at``. That is the point: a + quiet question that reappears every day trains the operator to skim, and + skimming is how a standing agenda stops being one. Nothing runs on a + schedule -- ``--due`` is a question you ask. + """ + now = utcnow() + with get_db() as session: + beat_id = resolve_beat_scope(session, beat_name) + + awaiting = ( + session.query(Prediction) + .filter( + Prediction.status == PREDICTION_STATUS_OPEN, + Prediction.due_by.is_not(None), + Prediction.due_by <= now, + prediction_scope_filter(session, beat_id), + ) + .order_by(Prediction.due_by.asc()) + .all() + ) + + open_questions = ( + session.query(Question) + .filter( + Question.status == QUESTION_STATUS_OPEN, + question_scope_filter(session, beat_id), + ) + .order_by(Question.first_asked_at.asc()) + .all() + ) + # Each question is tested against its *own* interval, not a global + # rhythm: two questions on different cadences come due independently. + due_questions = [q for q in open_questions if _question_is_due(q, now)] + uncadenced = sum(1 for q in open_questions if not q.cadence) + + click.echo(header(f"DUE NOW ({scope_label(beat_name)})")) + click.echo(muted(f"As of {now.date().isoformat()}. Nothing here runs on its own.")) + click.echo("=" * 80) + click.echo() + + due_predictions_header(len(awaiting)) + if not awaiting: + click.echo(muted(" Nothing of yours has come due.")) + click.echo() + else: + for p in awaiting: + render_due_prediction(p, now) + + click.echo(header(f"QUESTIONS DUE FOR REVIEW ({len(due_questions)})")) + click.echo(muted("Each question against its own cadence, not a shared rhythm.")) + click.echo() + if not due_questions: + click.echo(muted(" No question's review interval has elapsed.")) + click.echo() + else: + for q in due_questions: + _render_due_question(session, q, now, beat_id) + + # The stamp happens whether or not anything moved. + for q in due_questions: + q.last_reviewed_at = now + session.commit() + + if due_questions: + ids = ", ".join(f"Q{q.id}" for q in due_questions) + click.echo(success(f"Stamped as reviewed: {ids}.")) + click.echo( + muted("Reviewed means looked at, not resolved. They return at their next interval.") + ) + click.echo() + + if uncadenced: + click.echo( + muted( + f"{uncadenced} open question(s) carry no cadence and never surface " + "here. A cadence is your read on how fast a subject moves, so only " + "questions you declared with 'questions add' have one." + ) + ) + + +def _question_is_due(q: Question, now: datetime) -> bool: + """Whether one question's own interval has elapsed. A bad stored cadence is not due.""" + try: + return is_due(q.cadence, q.last_reviewed_at, q.first_asked_at, now) + except InvalidCadence: + return False + + +def _render_due_question(session, q: Question, now: datetime, beat_id: int | None) -> None: + """A due question plus whatever open claims are already staked on it.""" + try: + phrase = describe_next_review(q.cadence, q.last_reviewed_at, q.first_asked_at, now) + except InvalidCadence as exc: + phrase = f"unreadable cadence ({exc.raw})" + last = q.last_reviewed_at.date().isoformat() if q.last_reviewed_at else "never" + click.echo( + f" {accent(f'Q{q.id}')} {muted(f'cadence {q.cadence} | {phrase} | last reviewed {last}')}" + ) + click.echo(f" {q.text}") + + open_claims = ( + session.query(Prediction) + .filter( + Prediction.question_id == q.id, + Prediction.status == PREDICTION_STATUS_OPEN, + prediction_scope_filter(session, beat_id), + ) + .order_by(Prediction.made_at.asc()) + .all() + ) + if not open_claims: + click.echo( + muted(f' No open claim. Stake one: predict {q.id} "..." --by DATE --confidence N') + ) + for p in open_claims: + due = p.due_by.date().isoformat() if p.due_by else "no date" + conf = f"{p.confidence:.0%}" if p.confidence is not None else "no confidence" + click.echo(muted(f" P{p.id} [{p.author}] {conf}, due {due}: {p.observable_text}")) + click.echo() + + def _question_tag(question: Question | None) -> str: if question is None: return "" diff --git a/src/cli/predictions.py b/src/cli/predictions.py index 4445add..b80ff95 100644 --- a/src/cli/predictions.py +++ b/src/cli/predictions.py @@ -10,6 +10,10 @@ from ..context.beat_scope import prediction_scope_filter from ..database.connection import get_db from ..database.models import ( + PREDICTION_AUTHOR_MODEL, + PREDICTION_AUTHOR_OPERATOR, + PREDICTION_OUTCOME_NO, + PREDICTION_OUTCOME_YES, PREDICTION_STATUS_CONTRADICTED, PREDICTION_STATUS_EXPIRED, PREDICTION_STATUS_OPEN, @@ -33,16 +37,38 @@ def _print_predictions(rows, show_resolution: bool): age_days = (now - p.made_at).days question_text = p.question.text if p.question else "(unknown question)" click.echo( - f"{accent(f'P{p.id}')} {muted(f'made {p.made_at.date().isoformat()} ({age_days}d ago)')}" + f"{accent(f'P{p.id}')} [{p.author}] " + f"{muted(f'made {p.made_at.date().isoformat()} ({age_days}d ago)')}" ) click.echo(f" Observable: {p.observable_text}") click.echo(f" Trigger: {muted(p.trigger_condition)}") + if p.due_by is not None or p.confidence is not None: + due = p.due_by.date().isoformat() if p.due_by else "no date" + conf = f"{p.confidence:.0%}" if p.confidence is not None else "no confidence" + click.echo(muted(f" Staked at {conf}, resolving {due}")) click.echo(f" Bears on: {muted(question_text)}") + if show_resolution and p.outcome: + click.echo(muted(f" Outcome: {p.outcome} (recorded {_resolved_stamp(p)})")) if show_resolution and p.resolution_note: click.echo(f" {p.resolution_note}") click.echo() +def _resolved_stamp(p: Prediction) -> str: + """Resolution date, and how it sat against the due date.""" + if p.resolved_at is None: + return "date unknown" + stamp = p.resolved_at.date().isoformat() + if p.due_by is None: + return stamp + delta = (p.resolved_at.date() - p.due_by.date()).days + if delta > 0: + return f"{stamp}, {delta}d after it came due" + if delta < 0: + return f"{stamp}, {-delta}d before it came due" + return f"{stamp}, on the day" + + @predictions_command.command(name="open") @click.option("--limit", "-n", type=int, default=50) @beat_option @@ -143,6 +169,11 @@ def track_record(days, beat_name): only meaningful for one ledger. Mixing a beat's resolved observables into your own hit rate would corrupt the one figure the tool exists to be honest about. + + Split by author since 2026-08-27 (backlog task 011): the calibration figure + counts operator predictions only. Model predictions are reported under + their own heading and are never mixed in -- a track record blending the + two measures nothing. """ cutoff = utcnow() - timedelta(days=days) with get_db() as session: @@ -151,36 +182,95 @@ def track_record(days, beat_name): Prediction.made_at >= cutoff, prediction_scope_filter(session, beat_id), ) - total = window.count() - if total == 0: + if window.count() == 0: click.echo( muted(f"No predictions made in the last {days} days in {scope_label(beat_name)}.") ) return - counts = { - "open": window.filter(Prediction.status == PREDICTION_STATUS_OPEN).count(), - "triggered": window.filter(Prediction.status == PREDICTION_STATUS_TRIGGERED).count(), - "contradicted": window.filter( - Prediction.status == PREDICTION_STATUS_CONTRADICTED - ).count(), - "expired": window.filter(Prediction.status == PREDICTION_STATUS_EXPIRED).count(), - } - resolved = counts["triggered"] + counts["contradicted"] - click.echo(header(f"CALIBRATION RECORD (last {days} days, {scope_label(beat_name)})")) click.echo("=" * 80) - click.echo(f" Predictions made: {total}") - click.echo(f" {success('Triggered')}: {counts['triggered']}") - click.echo(f" {warning('Contradicted')}: {counts['contradicted']}") - click.echo(f" {muted('Still open')}: {counts['open']}") - click.echo(f" {muted('Expired (no signal)')}: {counts['expired']}") click.echo() - if resolved: - hit_rate = counts["triggered"] / resolved - click.echo( - f" Of {resolved} resolved predictions, " - f"{hit_rate:.0%} were triggered rather than contradicted." + _render_operator_record(window) + click.echo() + _render_model_record(window) + + +def _status_counts(window) -> dict[str, int]: + """Status tally for one slice of the ledger.""" + return { + "open": window.filter(Prediction.status == PREDICTION_STATUS_OPEN).count(), + "triggered": window.filter(Prediction.status == PREDICTION_STATUS_TRIGGERED).count(), + "contradicted": window.filter(Prediction.status == PREDICTION_STATUS_CONTRADICTED).count(), + "expired": window.filter(Prediction.status == PREDICTION_STATUS_EXPIRED).count(), + } + + +def _render_operator_record(window) -> None: + """ + The calibration figure. Operator predictions only. + + A plain hit rate on purpose -- how often the claim, as written, came true. + No weighting or Brier-style aggregation; that is a later decision once + there is real data to aggregate. + """ + mine = window.filter(Prediction.author == PREDICTION_AUTHOR_OPERATOR) + total = mine.count() + + click.echo(header("YOUR CALIBRATION (operator predictions only)")) + if total == 0: + click.echo( + muted( + " You have staked nothing in this window. The calibration figure " + 'counts only claims you made: \'predict "..." ' + "--by DATE --confidence N'." ) - else: - click.echo(muted(" No predictions resolved yet in this window.")) + ) + return + + right = mine.filter(Prediction.outcome == PREDICTION_OUTCOME_YES).count() + wrong = mine.filter(Prediction.outcome == PREDICTION_OUTCOME_NO).count() + counts = _status_counts(mine) + resolved = right + wrong + + click.echo(f" Claims staked: {total}") + click.echo(f" {success('Came true')}: {right}") + click.echo(f" {warning('Did not')}: {wrong}") + click.echo(f" {muted('Still open')}: {counts['open']}") + if counts["expired"]: + click.echo(f" {muted('Expired unjudged')}: {counts['expired']}") + click.echo() + if resolved: + click.echo(f" Hit rate: {right}/{resolved} = {right / resolved:.0%}.") + else: + click.echo(muted(" Nothing of yours has been resolved yet in this window.")) + + +def _render_model_record(window) -> None: + """ + The model's predictions, reported separately and never folded into the + figure above. They stay in the ledger as prompts -- suggestions about what + is worth holding an opinion on -- not as claims anyone staked. + """ + theirs = window.filter(Prediction.author == PREDICTION_AUTHOR_MODEL) + total = theirs.count() + + click.echo(header("MODEL PREDICTIONS (not counted toward your calibration)")) + if total == 0: + click.echo(muted(" The model made none in this window.")) + return + + counts = _status_counts(theirs) + click.echo( + muted( + f" {total} made | {counts['triggered']} triggered | " + f"{counts['contradicted']} contradicted | {counts['open']} open | " + f"{counts['expired']} expired" + ) + ) + click.echo( + muted( + " Reported for transparency only. These are the tool's own " + "observables, not your judgement." + ) + ) diff --git a/src/cli/questions.py b/src/cli/questions.py index ec8ce7c..107fb18 100644 --- a/src/cli/questions.py +++ b/src/cli/questions.py @@ -5,6 +5,7 @@ import click from ..context.beat_scope import owning_beat_names, question_scope_filter +from ..context.question_matcher import normalize_question from ..database.connection import get_db from ..database.models import ( QUESTION_STATUS_OPEN, @@ -14,6 +15,12 @@ QuestionSituation, ) from ..utils import utcnow +from ..utils.cadence import ( + CADENCE_EXAMPLES, + InvalidCadence, + describe_next_review, + normalize_cadence, +) from .colors import accent, header, muted, success, warning from .scope import beat_option, resolve_beat_scope, scope_label @@ -63,11 +70,70 @@ def list_questions(status, limit, beat_name): f"{muted(f'asked {q.first_asked_at.date().isoformat()} ({age_days}d ago)')}" ) click.echo(f" {q.text}") + prefix = f"cadence {q.cadence} | " if q.cadence else "" + click.echo(muted(f" {prefix}{_review_phrase(q, now)}")) if q.previous_question_id: click.echo(muted(f" (follows Q{q.previous_question_id})")) click.echo() +def _review_phrase(q: Question, now) -> str: + """Time-until-next-review for one question, tolerant of a bad stored cadence.""" + try: + return describe_next_review(q.cadence, q.last_reviewed_at, q.first_asked_at, now) + except InvalidCadence as exc: + return f"unreadable cadence ({exc.raw})" + + +@questions_command.command(name="add") +@click.argument("text") +@click.option( + "--cadence", + "cadence", + required=True, + metavar="INTERVAL", + help=( + "How often this question is worth re-examining, e.g. " + f"{', '.join(CADENCE_EXAMPLES)}. Required: the interval is your read on " + "how fast the subject moves, so the tool will not guess one." + ), +) +def add_question(text, cadence): + """ + Declare a question you want carried, at your own review cadence. + + Lands in your own ledger. A beat's agenda is declared in the beat config, + not here. + """ + claim = (text or "").strip() + if not claim: + raise click.UsageError("A question needs text. Nothing was stored.") + + try: + interval = normalize_cadence(cadence) + except InvalidCadence as exc: + raise click.UsageError(f"{exc} Nothing was stored.") + + with get_db() as session: + q = Question( + text=claim, + normalized_text=normalize_question(claim), + first_asked_at=utcnow(), + status=QUESTION_STATUS_OPEN, + cadence=interval, + # Never reviewed: the first interval counts from now, so a question + # declared today at 90d is not due today. + last_reviewed_at=None, + ) + session.add(q) + session.commit() + click.echo(success(f"Added question Q{q.id} at a {interval} review cadence.")) + click.echo(muted(f" {claim}")) + click.echo( + muted(f' Stake a claim on it with: predict {q.id} "..." --by DATE --confidence N') + ) + + @questions_command.command(name="show") @click.argument("question_id", type=int) def show_question(question_id): @@ -91,6 +157,14 @@ def show_question(question_id): click.echo(muted(f"Ledger: {', '.join(owners) if owners else 'yours (no beat)'}")) click.echo(muted(f"First asked: {q.first_asked_at.isoformat()}")) click.echo(muted(f"Status: {q.status}")) + click.echo( + muted( + f"Cadence: {q.cadence or 'none'} | " + f"{_review_phrase(q, utcnow())} | " + f"last reviewed: " + f"{q.last_reviewed_at.date().isoformat() if q.last_reviewed_at else 'never'}" + ) + ) if q.resolved_at: click.echo(muted(f"Resolved: {q.resolved_at.isoformat()}")) if q.resolution_note: diff --git a/src/cli/stake.py b/src/cli/stake.py new file mode 100644 index 0000000..483a79d --- /dev/null +++ b/src/cli/stake.py @@ -0,0 +1,238 @@ +""" +The operator's side of the ledger: stake a claim, then record the outcome. + +Every other ledger command reads. These two write, and they are the reason the +graph is a calibration instrument rather than a reading list -- the thing being +calibrated is the operator's judgment, and until these existed the operator had +no way to stake anything. + +Both commands are pure local database work: no API key, no network call. + +Added 2026-08-27 for backlog task 011. +""" + +from datetime import datetime + +import click + +from ..database.connection import get_db +from ..database.models import ( + PREDICTION_AUTHOR_OPERATOR, + PREDICTION_OUTCOME_NO, + PREDICTION_OUTCOME_YES, + PREDICTION_STATUS_CONTRADICTED, + PREDICTION_STATUS_OPEN, + PREDICTION_STATUS_TRIGGERED, + QUESTION_STATUS_OPEN, + Prediction, + Question, +) +from ..utils import utcnow +from .colors import accent, header, muted, success, warning + +# The failure this command exists to prevent, stated once so both the help text +# and the rejection message can quote it. +_WHY_DATE_REQUIRED = ( + "A claim with no resolution date can never come due, so it can never be " + "graded. That is how the ledger accumulated 25 predictions phrased " + "'X would signal Y' and graded none of them." +) + +_WHY_CONFIDENCE_REQUIRED = ( + "A stake with an unstated confidence is a non-commitment in a different " + "costume. There is deliberately no default." +) + +# yes/no is the operator's verdict; the status vocabulary the model's check +# pass already uses is reused so `predictions triggered|contradicted` and +# `forecast` keep working unchanged rather than growing a parallel view. +_OUTCOME_STATUS = { + PREDICTION_OUTCOME_YES: PREDICTION_STATUS_TRIGGERED, + PREDICTION_OUTCOME_NO: PREDICTION_STATUS_CONTRADICTED, +} + + +@click.command(name="predict") +@click.argument("question_id", type=int) +@click.argument("claim") +@click.option( + "--by", + "due_by", + type=click.DateTime(formats=["%Y-%m-%d"]), + default=None, + metavar="YYYY-MM-DD", + help=f"Required. When this claim resolves. {_WHY_DATE_REQUIRED}", +) +@click.option( + "--confidence", + "confidence", + type=float, + default=None, + metavar="0.0-1.0", + help=f"Required. Your stated probability. {_WHY_CONFIDENCE_REQUIRED}", +) +def predict_command(question_id, claim, due_by, confidence): + """ + Stake a falsifiable claim on a question, with a date and a confidence. + + Both --by and --confidence are rejected at entry when missing. Nothing is + stored on a rejection. + """ + text = (claim or "").strip() + if not text: + raise click.UsageError("A prediction needs a claim. Nothing was stored.") + + if due_by is None: + raise click.UsageError( + f"--by is required. {_WHY_DATE_REQUIRED} " + "Give a resolution date, e.g. --by 2026-12-31. Nothing was stored." + ) + + if confidence is None: + raise click.UsageError( + f"--confidence is required. {_WHY_CONFIDENCE_REQUIRED} " + "Give a probability, e.g. --confidence 0.7. Nothing was stored." + ) + + if not 0.0 <= confidence <= 1.0: + raise click.UsageError( + f"--confidence must be between 0.0 and 1.0; got {confidence}. Nothing was stored." + ) + + now = utcnow() + if due_by.date() < now.date(): + raise click.UsageError( + f"--by {due_by.date().isoformat()} is in the past. A claim that was " + "already resolvable when you made it is not a forecast. Nothing was stored." + ) + + with get_db() as session: + question = session.query(Question).filter_by(id=question_id).first() + if question is None: + raise click.UsageError( + f"Question Q{question_id} not found. Declare it first with " + "'questions add'. Nothing was stored." + ) + if question.status != QUESTION_STATUS_OPEN: + raise click.UsageError( + f"Q{question_id} is {question.status}, not open. Nothing was stored." + ) + + prediction = Prediction( + question_id=question.id, + observable_text=text, + # The trigger for an operator claim is the operator, on the date + # they named. Recorded explicitly so the column keeps meaning the + # same thing it does for model predictions: what makes this + # gradeable. + trigger_condition=(f"Operator judgement on or after {due_by.date().isoformat()}."), + made_at=now, + made_in_synthesis_id=None, + author=PREDICTION_AUTHOR_OPERATOR, + due_by=due_by, + confidence=float(confidence), + status=PREDICTION_STATUS_OPEN, + ) + session.add(prediction) + session.commit() + + days_out = (due_by.date() - now.date()).days + click.echo( + success( + f"Staked P{prediction.id} on Q{question.id} at " + f"{confidence:.0%}, resolving {due_by.date().isoformat()} ({days_out}d out)." + ) + ) + click.echo(muted(f" Claim: {text}")) + click.echo(muted(f" Bears on: {question.text}")) + click.echo(muted(f" Record the outcome with: resolve {prediction.id} --outcome yes|no")) + + +@click.command(name="resolve") +@click.argument("prediction_id", type=int) +@click.option( + "--outcome", + type=click.Choice([PREDICTION_OUTCOME_YES, PREDICTION_OUTCOME_NO]), + required=True, + help="Did the claim, as written, come true?", +) +@click.option( + "--note", + required=True, + help="What settled it. Required: an outcome with no reason is not calibration data.", +) +def resolve_command(prediction_id, outcome, note): + """ + Record the outcome of a prediction. You resolve; the tool does not. + + Auto-resolving from coverage is deliberately absent: a tool that grades its + operator's calls using the corpus that produced them is measuring agreement + with itself. + """ + now = utcnow() + with get_db() as session: + prediction = session.query(Prediction).filter_by(id=prediction_id).first() + if prediction is None: + raise click.UsageError(f"Prediction P{prediction_id} not found. Nothing was changed.") + + if prediction.status != PREDICTION_STATUS_OPEN: + when = ( + prediction.resolved_at.date().isoformat() if prediction.resolved_at else "unknown" + ) + verdict = f" as '{prediction.outcome}'" if prediction.outcome else "" + raise click.UsageError( + f"P{prediction.id} is already {prediction.status}{verdict} " + f"(recorded {when}). A resolved prediction is not editable. " + "Nothing was changed." + ) + + prediction.outcome = outcome + prediction.status = _OUTCOME_STATUS[outcome] + # Separate from due_by on purpose: how late you graded it is itself + # calibration data. + prediction.resolved_at = now + prediction.resolution_note = note + question_text = prediction.question.text if prediction.question else None + session.commit() + + click.echo(success(f"P{prediction.id} resolved: {outcome}.")) + click.echo(muted(f" Claim: {prediction.observable_text}")) + if question_text: + click.echo(muted(f" Bears on: {question_text}")) + click.echo(muted(f" Note: {note}")) + click.echo(muted(f" {_timing_phrase(prediction.due_by, now)}")) + + +def _timing_phrase(due_by: datetime | None, resolved_at: datetime) -> str: + """How the resolution date sits against the due date.""" + if due_by is None: + return f"Resolved {resolved_at.date().isoformat()} (no due date on record)." + delta = (resolved_at.date() - due_by.date()).days + stamp = f"Due {due_by.date().isoformat()}, resolved {resolved_at.date().isoformat()}" + if delta > 0: + return f"{stamp} -- {delta}d late." + if delta < 0: + return f"{stamp} -- {-delta}d early, before it came due." + return f"{stamp} -- on the day." + + +def render_due_prediction(p: Prediction, now: datetime) -> None: + """One line-block for a prediction awaiting the operator's verdict.""" + overdue = (now.date() - p.due_by.date()).days if p.due_by else None + when = p.due_by.date().isoformat() if p.due_by else "no due date" + lateness = f" ({overdue}d ago)" if overdue else "" + click.echo(f" {accent(f'P{p.id}')} {warning('due ' + when)}{muted(lateness)}") + click.echo(f" {p.observable_text}") + if p.confidence is not None: + click.echo(muted(f" You said: {p.confidence:.0%}")) + click.echo(muted(f" resolve {p.id} --outcome yes|no --note '...'")) + click.echo() + + +def due_predictions_header(count: int) -> None: + """Heading for the awaiting-verdict block, shared by `forecast --due`.""" + click.echo(header(f"AWAITING YOUR VERDICT ({count})")) + click.echo( + muted("Your claims whose resolution date has passed. You grade these; nothing else does.") + ) + click.echo() diff --git a/src/database/migrations/add_calibration_loop.py b/src/database/migrations/add_calibration_loop.py new file mode 100644 index 0000000..0cc3a8e --- /dev/null +++ b/src/database/migrations/add_calibration_loop.py @@ -0,0 +1,261 @@ +""" +Migration: Add the operator calibration loop (2026-08-27, backlog task 011) + +Two tables change, both preserving every existing row: + +``questions`` gains ``cadence`` and ``last_reviewed_at``. Purely additive -- +two nullable columns via ALTER TABLE. Every pre-existing question keeps a null +cadence, which is correct: it was surfaced by the model from coverage, not put +on a review schedule by the operator, so ``forecast --due`` never raises it. + +``predictions`` gains ``author``, ``due_by``, ``confidence`` and ``outcome``, +and relaxes ``made_in_synthesis_id`` to nullable so a claim the operator stakes +from the CLI -- which has no synthesis behind it -- can be stored at all. +SQLite cannot drop a NOT NULL constraint in place, so this half is the standard +SQLite table rebuild: rename, recreate from the model, copy every row, drop the +old table. Every existing prediction backfills to ``author='model'``, which is +what all of them were. + +Reversible. ``downgrade`` restores both tables to their pre-011 shape, and +refuses rather than destroying data if any operator prediction exists: an +operator-staked claim has no representation in the old schema, so silently +dropping or relabelling one would be a lie about who said it. +""" + +from sqlalchemy import inspect, text + +from src.database.connection import engine +from src.database.models import Prediction, Question + +# The pre-011 shape of `predictions`, needed verbatim by downgrade(). Kept as a +# literal rather than derived from the model, because the model has moved on. +_PREDICTIONS_PRE_011_DDL = """ +CREATE TABLE predictions ( + id INTEGER NOT NULL, + question_id INTEGER NOT NULL, + observable_text TEXT NOT NULL, + trigger_condition TEXT NOT NULL, + made_at DATETIME NOT NULL, + made_in_synthesis_id INTEGER NOT NULL, + status VARCHAR(20) NOT NULL, + resolved_at DATETIME, + resolution_note TEXT, + PRIMARY KEY (id), + FOREIGN KEY(question_id) REFERENCES questions (id), + FOREIGN KEY(made_in_synthesis_id) REFERENCES narrative_syntheses (id) +) +""" + +_PREDICTIONS_PRE_011_COLUMNS = [ + "id", + "question_id", + "observable_text", + "trigger_condition", + "made_at", + "made_in_synthesis_id", + "status", + "resolved_at", + "resolution_note", +] + +_PREDICTIONS_PRE_011_INDEXES = [ + "CREATE INDEX idx_prediction_status ON predictions (status)", + "CREATE INDEX idx_prediction_question ON predictions (question_id)", + "CREATE INDEX idx_prediction_made_at ON predictions (made_at)", +] + +# Every index on `predictions`, old and new. Dropped before a rebuild so the +# names are free: SQLite keeps indexes attached to a table across a RENAME, +# under their original names, which would collide on recreate. +_ALL_PREDICTION_INDEXES = [ + "idx_prediction_status", + "idx_prediction_question", + "idx_prediction_made_at", + "idx_prediction_author", + "idx_prediction_due_by", +] + +_QUESTION_COLUMNS = { + "cadence": "VARCHAR(20)", + "last_reviewed_at": "DATETIME", +} + + +def _column_names(conn, table: str) -> set[str]: + return {col["name"] for col in inspect(conn).get_columns(table)} + + +def _has_table(conn, table: str) -> bool: + return inspect(conn).has_table(table) + + +def _require_sqlite(conn) -> None: + """ + The rebuild half is SQLite-specific. + + Failing loudly beats emitting DDL that silently does the wrong thing on + another engine. InsightWeaver has only ever run on SQLite. + """ + if conn.dialect.name != "sqlite": + raise RuntimeError( + f"This migration rebuilds the predictions table using SQLite semantics; " + f"the configured database is '{conn.dialect.name}'. Port it before running." + ) + + +def upgrade(): + """Add the cadence and authorship columns, preserving every row.""" + with engine.begin() as conn: + _require_sqlite(conn) + + print("Adding review cadence to questions...") + if not _has_table(conn, "questions"): + Question.__table__.create(conn, checkfirst=True) + print(" questions table created") + else: + existing = _column_names(conn, "questions") + for name, sql_type in _QUESTION_COLUMNS.items(): + if name in existing: + print(f" questions.{name} already present, skipping") + continue + conn.execute(text(f"ALTER TABLE questions ADD COLUMN {name} {sql_type}")) + print(f" questions.{name} added") + + print("\nAdding authorship, due date and outcome to predictions...") + if not _has_table(conn, "predictions"): + Prediction.__table__.create(conn, checkfirst=True) + print(" predictions table created") + elif "author" in _column_names(conn, "predictions"): + print(" predictions already carries author, skipping rebuild") + else: + _rebuild_predictions_forward(conn) + + print("\nCalibration loop migration completed.") + + +def _rebuild_predictions_forward(conn) -> None: + """ + SQLite table rebuild: relax made_in_synthesis_id and add the new columns. + + Row count is asserted on both sides. A rebuild that quietly lost rows would + corrupt the one ledger this task exists to make trustworthy. + """ + before = conn.execute(text("SELECT COUNT(*) FROM predictions")).scalar_one() + print(f" rebuilding predictions ({before} row(s) to carry over)") + + for index_name in _ALL_PREDICTION_INDEXES: + conn.execute(text(f"DROP INDEX IF EXISTS {index_name}")) + conn.execute(text("ALTER TABLE predictions RENAME TO predictions_pre_011")) + + Prediction.__table__.create(conn, checkfirst=False) + + carried = ", ".join(_PREDICTIONS_PRE_011_COLUMNS) + conn.execute( + text( + f"INSERT INTO predictions ({carried}, author) " + f"SELECT {carried}, 'model' FROM predictions_pre_011" + ) + ) + + after = conn.execute(text("SELECT COUNT(*) FROM predictions")).scalar_one() + if after != before: + raise RuntimeError( + f"Rebuild carried {after} of {before} predictions. Rolling back; " + "the database is unchanged." + ) + + conn.execute(text("DROP TABLE predictions_pre_011")) + print(f" {after} prediction(s) backfilled to author='model'") + + +def downgrade(): + """Restore the pre-011 shape of both tables.""" + with engine.begin() as conn: + _require_sqlite(conn) + + print("Reverting predictions...") + if not _has_table(conn, "predictions"): + print(" no predictions table, nothing to revert") + elif "author" not in _column_names(conn, "predictions"): + print(" predictions already at the pre-011 shape, skipping") + else: + _rebuild_predictions_back(conn) + + print("\nReverting questions...") + if not _has_table(conn, "questions"): + print(" no questions table, nothing to revert") + else: + existing = _column_names(conn, "questions") + for name in _QUESTION_COLUMNS: + if name not in existing: + print(f" questions.{name} already absent, skipping") + continue + conn.execute(text(f"ALTER TABLE questions DROP COLUMN {name}")) + print(f" questions.{name} dropped") + + print("\nCalibration loop downgrade completed.") + + +def _rebuild_predictions_back(conn) -> None: + """ + Rebuild predictions to the pre-011 shape, or refuse. + + An operator prediction has no synthesis and no author column to live in + once this runs, so it cannot survive the revert. Refusing is the honest + answer; quietly relabelling it 'model' would put words in the operator's + mouth, and dropping it would delete the record silently. + """ + stakes = conn.execute( + text("SELECT COUNT(*) FROM predictions WHERE author = 'operator'") + ).scalar_one() + if stakes: + raise RuntimeError( + f"{stakes} operator prediction(s) are in the ledger. The pre-011 schema " + "has nowhere to record who staked a claim or when it resolves, so this " + "downgrade would silently misattribute them. Export or delete them " + "first if you really mean to revert. Nothing was changed." + ) + + orphans = conn.execute( + text("SELECT COUNT(*) FROM predictions WHERE made_in_synthesis_id IS NULL") + ).scalar_one() + if orphans: + raise RuntimeError( + f"{orphans} prediction(s) have no synthesis, which the pre-011 schema " + "requires. Nothing was changed." + ) + + before = conn.execute(text("SELECT COUNT(*) FROM predictions")).scalar_one() + print(f" rebuilding predictions ({before} row(s) to carry over)") + + for index_name in _ALL_PREDICTION_INDEXES: + conn.execute(text(f"DROP INDEX IF EXISTS {index_name}")) + conn.execute(text("ALTER TABLE predictions RENAME TO predictions_post_011")) + + conn.execute(text(_PREDICTIONS_PRE_011_DDL)) + for statement in _PREDICTIONS_PRE_011_INDEXES: + conn.execute(text(statement)) + + carried = ", ".join(_PREDICTIONS_PRE_011_COLUMNS) + conn.execute( + text(f"INSERT INTO predictions ({carried}) SELECT {carried} FROM predictions_post_011") + ) + + after = conn.execute(text("SELECT COUNT(*) FROM predictions")).scalar_one() + if after != before: + raise RuntimeError( + f"Revert carried {after} of {before} predictions. Rolling back; " + "the database is unchanged." + ) + + conn.execute(text("DROP TABLE predictions_post_011")) + print(f" {after} prediction(s) restored") + + +if __name__ == "__main__": + import sys + + if len(sys.argv) > 1 and sys.argv[1] == "down": + downgrade() + else: + upgrade() diff --git a/src/database/models.py b/src/database/models.py index 146de18..4f2ec7d 100644 --- a/src/database/models.py +++ b/src/database/models.py @@ -349,6 +349,25 @@ class Question(Base): # Secondary questions are tracked but rendered quieter in the brief. is_primary = Column(Boolean, default=True, nullable=False) + # Review interval, e.g. "7d", "90d", "1y" (2026-08-27, backlog task 011). + # A cadence is NOT a deadline: it says how often this question is worth + # re-examining, while a Prediction's due_by says when a specific claim + # resolves. A question reviewed quarterly can hold a claim resolving in + # three weeks. + # + # Nullable, and null means "not on any review schedule". Every question + # that predates this column is model-emergent -- the graph noticed it in + # coverage, the operator never put it on an agenda -- so `forecast --due` + # correctly never surfaces it. A cadence is the operator's read on how fast + # a subject moves, so only the operator can set one, via `questions add`. + cadence = Column(String(20), nullable=True) + + # When the operator last reviewed this question. Stamped by `forecast --due` + # whether or not anything moved: a quiet question that reappears every day + # trains the operator to skim. Null means never reviewed, in which case + # first_asked_at is the baseline the interval counts from. + last_reviewed_at = Column(DateTime, nullable=True) + situation_links = relationship("QuestionSituation", back_populates="question") __table_args__ = ( @@ -402,6 +421,18 @@ class QuestionSituation(Base): # that far out is unlikely to still bear on the original observable. PREDICTION_EXPIRY_DAYS = 90 +# Who staked the claim (2026-08-27, backlog task 011). The calibration figure +# counts operator predictions only; model predictions stay in the ledger as +# prompts -- suggestions about what is worth holding an opinion on -- and are +# reported under their own heading, never mixed in. +PREDICTION_AUTHOR_OPERATOR = "operator" +PREDICTION_AUTHOR_MODEL = "model" + +# What the operator judged when resolving. Recorded alongside status so the +# verdict survives independently of the status vocabulary the check pass uses. +PREDICTION_OUTCOME_YES = "yes" +PREDICTION_OUTCOME_NO = "no" + class Prediction(Base): """ @@ -422,9 +453,35 @@ class Prediction(Base): trigger_condition = Column(Text, nullable=False) made_at = Column(DateTime, default=utcnow, nullable=False) - made_in_synthesis_id = Column(Integer, ForeignKey("narrative_syntheses.id"), nullable=False) + # Nullable since 2026-08-27 (backlog task 011): an operator stakes a claim + # from the CLI, with no synthesis behind it. Model predictions still carry + # the synthesis that produced them. + made_in_synthesis_id = Column(Integer, ForeignKey("narrative_syntheses.id"), nullable=True) + + # operator | model. Not nullable, because "who said this" is the one fact + # the calibration figure depends on; every pre-existing row backfills to + # 'model', which is what all 33 of them were. + author = Column(String(20), nullable=False, default=PREDICTION_AUTHOR_MODEL) + + # When this specific claim resolves. Required at entry for an operator + # prediction -- a claim with no resolution date cannot come due and so + # cannot be graded, which is exactly how the ledger ended up with 19 + # predictions that expired unjudged. Null on model predictions, which are + # aged out by PREDICTION_EXPIRY_DAYS instead. + due_by = Column(DateTime, nullable=True) + + # Stated probability, 0.0-1.0. Required at entry for an operator + # prediction and deliberately given no default: an unstated confidence is + # a non-commitment in a different costume. + confidence = Column(Float, nullable=True) status = Column(String(20), nullable=False, default=PREDICTION_STATUS_OPEN) + + # yes | no -- the operator's verdict on the claim as written. + outcome = Column(String(20), nullable=True) + + # When the verdict was recorded. Deliberately separate from due_by: + # resolving three months after the due date is itself calibration data. resolved_at = Column(DateTime, nullable=True) resolution_note = Column(Text, nullable=True) @@ -434,6 +491,8 @@ class Prediction(Base): Index("idx_prediction_status", "status"), Index("idx_prediction_question", "question_id"), Index("idx_prediction_made_at", "made_at"), + Index("idx_prediction_author", "author"), + Index("idx_prediction_due_by", "due_by"), ) diff --git a/src/utils/cadence.py b/src/utils/cadence.py new file mode 100644 index 0000000..fc62652 --- /dev/null +++ b/src/utils/cadence.py @@ -0,0 +1,117 @@ +""" +Review cadences: how often a question is worth re-examining. + +A cadence is written the way an operator says it out loud -- ``7d``, ``90d``, +``1y`` -- and parsed here into a plain interval. It is **not** a deadline. A +question's cadence says how often to look at it again; a prediction's +``due_by`` says when a specific claim resolves. A question reviewed quarterly +can hold a claim that resolves in three weeks, and collapsing the two into one +field would destroy that nesting. + +Added 2026-08-27 for backlog task 011. +""" + +from __future__ import annotations + +import re +from datetime import datetime, timedelta + +# Units are calendar-approximate on purpose. A cadence is the operator's read +# on how fast a subject moves, not a contract -- "90d" and "3m" being the same +# interval is fine, and month-end arithmetic would add precision the concept +# does not have. +_UNIT_DAYS = {"d": 1, "w": 7, "m": 30, "y": 365} + +_CADENCE_RE = re.compile(r"^\s*(\d+)\s*([dwmy])\s*$", re.IGNORECASE) + +# The forms the help text advertises, in the order it advertises them. +CADENCE_EXAMPLES = ("7d", "30d", "90d", "1y") + + +class InvalidCadence(ValueError): + """Raised when a cadence string cannot be read as an interval.""" + + def __init__(self, raw: str) -> None: + self.raw = raw + super().__init__( + f"'{raw}' is not a review cadence. Write a whole number followed by " + f"d (days), w (weeks), m (months, 30d) or y (years, 365d) -- " + f"for example {', '.join(CADENCE_EXAMPLES)}." + ) + + +def parse_cadence(raw: str) -> timedelta: + """ + Turn ``7d`` / ``2w`` / ``3m`` / ``1y`` into an interval. + + Raises :class:`InvalidCadence` rather than defaulting. A cadence the tool + silently guessed at would not be the operator's read on the subject, which + is the only thing that makes the interval itself falsifiable. + """ + match = _CADENCE_RE.match(raw or "") + if not match: + raise InvalidCadence(raw) + amount = int(match.group(1)) + if amount <= 0: + raise InvalidCadence(raw) + return timedelta(days=amount * _UNIT_DAYS[match.group(2).lower()]) + + +def normalize_cadence(raw: str) -> str: + """Canonical spelling of a valid cadence: lowercased, whitespace stripped.""" + match = _CADENCE_RE.match(raw or "") + if not match: + raise InvalidCadence(raw) + if int(match.group(1)) <= 0: + raise InvalidCadence(raw) + return f"{int(match.group(1))}{match.group(2).lower()}" + + +def next_review_at( + cadence: str | None, + last_reviewed_at: datetime | None, + first_asked_at: datetime, +) -> datetime | None: + """ + When this question next comes up for review, or None if it has no cadence. + + The interval counts from the last review, or from when the question was + first asked if it has never been reviewed -- a question declared today at a + 90-day cadence is not due today. + """ + if not cadence: + return None + baseline = last_reviewed_at or first_asked_at + return baseline + parse_cadence(cadence) + + +def is_due( + cadence: str | None, + last_reviewed_at: datetime | None, + first_asked_at: datetime, + now: datetime, +) -> bool: + """Whether this question's own interval has elapsed since its last review.""" + due_at = next_review_at(cadence, last_reviewed_at, first_asked_at) + return due_at is not None and now >= due_at + + +def describe_next_review( + cadence: str | None, + last_reviewed_at: datetime | None, + first_asked_at: datetime, + now: datetime, +) -> str: + """ + A short human phrase for time-until-next-review, for ``questions list``. + + Says "no cadence" rather than inventing one, because a question with no + cadence is genuinely not on a review schedule. + """ + due_at = next_review_at(cadence, last_reviewed_at, first_asked_at) + if due_at is None: + return "no cadence" + if now >= due_at: + overdue = (now - due_at).days + return f"due now ({overdue}d overdue)" if overdue else "due now" + return f"next review in {max((due_at - now).days, 0)}d" diff --git a/tests/cli/test_calibration_loop.py b/tests/cli/test_calibration_loop.py new file mode 100644 index 0000000..29f60f9 --- /dev/null +++ b/tests/cli/test_calibration_loop.py @@ -0,0 +1,584 @@ +""" +Tests for the operator calibration loop (backlog task 011). + +The finding this feature answers: the tool generated 33 predictions and graded +zero, because 25 were phrased "X would signal Y" and only 3 carried a date. The +tests that matter most here are therefore the refusals -- a claim with no date +must not reach the database at all. +""" + +import socket +from contextlib import contextmanager +from datetime import timedelta +from unittest.mock import patch + +import pytest + +from src.cli.forecast import forecast_command +from src.cli.predictions import predictions_command +from src.cli.questions import questions_command +from src.cli.stake import predict_command, resolve_command +from src.database.models import ( + PREDICTION_AUTHOR_MODEL, + PREDICTION_AUTHOR_OPERATOR, + PREDICTION_STATUS_CONTRADICTED, + PREDICTION_STATUS_OPEN, + PREDICTION_STATUS_TRIGGERED, + QUESTION_STATUS_OPEN, + AnalysisRun, + NarrativeSynthesis, + Prediction, + Question, +) +from src.utils import utcnow + +# Every command that touches the loop reads the database through its own +# module's get_db, so all of them are redirected at once. +_DB_TARGETS = ( + "src.cli.questions.get_db", + "src.cli.stake.get_db", + "src.cli.forecast.get_db", + "src.cli.predictions.get_db", +) + + +@contextmanager +def use_session(session): + """Point every loop command at one test session.""" + + @contextmanager + def _ctx(): + try: + yield session + session.commit() + except Exception: + session.rollback() + raise + + with ( + patch(_DB_TARGETS[0], _ctx), + patch(_DB_TARGETS[1], _ctx), + patch(_DB_TARGETS[2], _ctx), + patch(_DB_TARGETS[3], _ctx), + ): + yield + + +def run(cli_runner, session, command, args): + with use_session(session): + return cli_runner.invoke(command, args) + + +def make_question(session, text, cadence, *, first_asked_days_ago=0, last_reviewed_days_ago=None): + q = Question( + text=text, + normalized_text=text.lower(), + first_asked_at=utcnow() - timedelta(days=first_asked_days_ago), + status=QUESTION_STATUS_OPEN, + cadence=cadence, + last_reviewed_at=( + None + if last_reviewed_days_ago is None + else utcnow() - timedelta(days=last_reviewed_days_ago) + ), + ) + session.add(q) + session.commit() + return q + + +def future_date(days): + return (utcnow() + timedelta(days=days)).date().isoformat() + + +class TestQuestionsAdd: + def test_declares_a_question_at_a_cadence(self, cli_runner, test_session): + result = run( + cli_runner, + test_session, + questions_command, + ["add", "Does CMMC Phase 2 slip past its statutory date?", "--cadence", "90d"], + ) + + assert result.exit_code == 0 + stored = test_session.query(Question).one() + assert stored.text == "Does CMMC Phase 2 slip past its statutory date?" + assert stored.cadence == "90d" + # Never reviewed: the first interval counts from when it was declared. + assert stored.last_reviewed_at is None + + def test_cadence_is_required(self, cli_runner, test_session): + result = run(cli_runner, test_session, questions_command, ["add", "Something"]) + + assert result.exit_code != 0 + assert test_session.query(Question).count() == 0 + + def test_unreadable_cadence_is_refused(self, cli_runner, test_session): + result = run( + cli_runner, test_session, questions_command, ["add", "Something", "--cadence", "soon"] + ) + + assert result.exit_code != 0 + assert "not a review cadence" in result.output + assert "Nothing was stored" in result.output + assert test_session.query(Question).count() == 0 + + def test_list_shows_cadence_and_time_until_next_review(self, cli_runner, test_session): + make_question(test_session, "Slow question", "90d", first_asked_days_ago=10) + + result = run(cli_runner, test_session, questions_command, ["list"]) + + assert result.exit_code == 0 + assert "cadence 90d" in result.output + assert "next review in 79d" in result.output + + +class TestDateIsRejectedAtEntry: + """ + The core requirement. A claim with no resolution date can never come due, + so it can never be graded -- which is how 19 predictions expired unjudged. + """ + + def test_missing_by_is_refused_and_nothing_is_stored(self, cli_runner, test_session): + q = make_question(test_session, "Does CMMC Phase 2 slip?", "90d") + + result = run( + cli_runner, + test_session, + predict_command, + [str(q.id), "Yes -- slips", "--confidence", "0.7"], + ) + + assert result.exit_code != 0 + assert "--by is required" in result.output + assert "Nothing was stored" in result.output + assert test_session.query(Prediction).count() == 0 + + def test_missing_confidence_is_refused_and_nothing_is_stored(self, cli_runner, test_session): + q = make_question(test_session, "Does CMMC Phase 2 slip?", "90d") + + result = run( + cli_runner, + test_session, + predict_command, + [str(q.id), "Yes -- slips", "--by", future_date(60)], + ) + + assert result.exit_code != 0 + assert "--confidence is required" in result.output + assert test_session.query(Prediction).count() == 0 + + def test_no_confidence_default_is_applied(self, cli_runner, test_session): + """A stake with an unstated confidence is a non-commitment in costume.""" + q = make_question(test_session, "Does CMMC Phase 2 slip?", "90d") + + run( + cli_runner, + test_session, + predict_command, + [str(q.id), "Yes -- slips", "--by", future_date(60)], + ) + + assert test_session.query(Prediction).count() == 0 + + def test_past_due_date_is_refused(self, cli_runner, test_session): + q = make_question(test_session, "Does CMMC Phase 2 slip?", "90d") + past = (utcnow() - timedelta(days=1)).date().isoformat() + + result = run( + cli_runner, + test_session, + predict_command, + [str(q.id), "Yes -- slips", "--by", past, "--confidence", "0.7"], + ) + + assert result.exit_code != 0 + assert "is in the past" in result.output + assert test_session.query(Prediction).count() == 0 + + def test_confidence_outside_zero_to_one_is_refused(self, cli_runner, test_session): + q = make_question(test_session, "Does CMMC Phase 2 slip?", "90d") + + result = run( + cli_runner, + test_session, + predict_command, + [str(q.id), "Yes -- slips", "--by", future_date(60), "--confidence", "7"], + ) + + assert result.exit_code != 0 + assert test_session.query(Prediction).count() == 0 + + def test_unknown_question_is_refused(self, cli_runner, test_session): + result = run( + cli_runner, + test_session, + predict_command, + ["999", "Yes", "--by", future_date(30), "--confidence", "0.5"], + ) + + assert result.exit_code != 0 + assert test_session.query(Prediction).count() == 0 + + +class TestStakingStores: + def test_a_complete_claim_is_stored_with_author_date_and_confidence( + self, cli_runner, test_session + ): + q = make_question(test_session, "Does CMMC Phase 2 slip?", "90d") + + result = run( + cli_runner, + test_session, + predict_command, + [str(q.id), "Yes -- slips", "--by", "2099-12-31", "--confidence", "0.7"], + ) + + assert result.exit_code == 0 + p = test_session.query(Prediction).one() + assert p.author == PREDICTION_AUTHOR_OPERATOR + assert p.observable_text == "Yes -- slips" + assert p.due_by.date().isoformat() == "2099-12-31" + assert p.confidence == pytest.approx(0.7) + assert p.status == PREDICTION_STATUS_OPEN + assert p.made_in_synthesis_id is None + assert p.outcome is None + + +class TestResolve: + def _staked(self, cli_runner, test_session, days_out=30): + q = make_question(test_session, "Does CMMC Phase 2 slip?", "90d") + run( + cli_runner, + test_session, + predict_command, + [str(q.id), "Yes -- slips", "--by", future_date(days_out), "--confidence", "0.7"], + ) + return q, test_session.query(Prediction).one() + + def test_resolving_before_the_due_date_is_allowed_and_says_so(self, cli_runner, test_session): + """ + The event can settle early. What matters is that the resolution date is + recorded separately from the due date, so grading early -- or three + months late -- is itself visible. + """ + _, p = self._staked(cli_runner, test_session, days_out=30) + + result = run( + cli_runner, + test_session, + resolve_command, + [str(p.id), "--outcome", "no", "--note", "DFARS class deviation published."], + ) + + assert result.exit_code == 0 + assert "before it came due" in result.output + test_session.refresh(p) + assert p.outcome == "no" + assert p.status == PREDICTION_STATUS_CONTRADICTED + assert p.resolved_at is not None + assert p.resolved_at.date() < p.due_by.date() + assert p.resolution_note == "DFARS class deviation published." + + def test_resolving_twice_is_refused(self, cli_runner, test_session): + _, p = self._staked(cli_runner, test_session) + run( + cli_runner, + test_session, + resolve_command, + [str(p.id), "--outcome", "yes", "--note", "First verdict."], + ) + + result = run( + cli_runner, + test_session, + resolve_command, + [str(p.id), "--outcome", "no", "--note", "Changed my mind."], + ) + + assert result.exit_code != 0 + assert "already triggered" in result.output + assert "not editable" in result.output + test_session.refresh(p) + assert p.outcome == "yes" + assert p.status == PREDICTION_STATUS_TRIGGERED + assert p.resolution_note == "First verdict." + + def test_note_is_required(self, cli_runner, test_session): + _, p = self._staked(cli_runner, test_session) + + result = run(cli_runner, test_session, resolve_command, [str(p.id), "--outcome", "yes"]) + + assert result.exit_code != 0 + test_session.refresh(p) + assert p.status == PREDICTION_STATUS_OPEN + + def test_unknown_prediction_is_refused(self, cli_runner, test_session): + result = run( + cli_runner, test_session, resolve_command, ["999", "--outcome", "yes", "--note", "x"] + ) + + assert result.exit_code != 0 + assert "not found" in result.output + + +class TestDueRespectsEachQuestionsOwnCadence: + def test_a_question_whose_cadence_has_not_elapsed_is_absent(self, cli_runner, test_session): + make_question(test_session, "Slow moving subject", "90d", first_asked_days_ago=10) + + result = run(cli_runner, test_session, forecast_command, ["--due"]) + + assert result.exit_code == 0 + assert "Slow moving subject" not in result.output + assert "QUESTIONS DUE FOR REVIEW (0)" in result.output + + def test_two_cadences_come_due_independently(self, cli_runner, test_session): + """ + The whole point of a per-question interval: at ten days elapsed the + weekly question is due and the quarterly one is not. + """ + fast = make_question(test_session, "Fast moving subject", "7d", first_asked_days_ago=10) + slow = make_question(test_session, "Slow moving subject", "90d", first_asked_days_ago=10) + + result = run(cli_runner, test_session, forecast_command, ["--due"]) + + assert result.exit_code == 0 + assert "Fast moving subject" in result.output + assert "Slow moving subject" not in result.output + assert "QUESTIONS DUE FOR REVIEW (1)" in result.output + + test_session.refresh(fast) + test_session.refresh(slow) + assert fast.last_reviewed_at is not None + assert slow.last_reviewed_at is None + + def test_the_slow_one_comes_due_on_its_own_schedule(self, cli_runner, test_session): + make_question(test_session, "Fast moving subject", "7d", first_asked_days_ago=100) + make_question(test_session, "Slow moving subject", "90d", first_asked_days_ago=100) + + result = run(cli_runner, test_session, forecast_command, ["--due"]) + + assert "Fast moving subject" in result.output + assert "Slow moving subject" in result.output + assert "QUESTIONS DUE FOR REVIEW (2)" in result.output + + def test_reviewing_stamps_even_when_nothing_moved(self, cli_runner, test_session): + q = make_question(test_session, "Quiet question", "7d", first_asked_days_ago=10) + + first = run(cli_runner, test_session, forecast_command, ["--due"]) + assert "Quiet question" in first.output + assert "Stamped as reviewed: Q" in first.output + + second = run(cli_runner, test_session, forecast_command, ["--due"]) + assert "Quiet question" not in second.output + assert "QUESTIONS DUE FOR REVIEW (0)" in second.output + + test_session.refresh(q) + assert q.last_reviewed_at is not None + + def test_a_question_with_no_cadence_never_surfaces(self, cli_runner, test_session): + make_question(test_session, "Model emergent question", None, first_asked_days_ago=400) + + result = run(cli_runner, test_session, forecast_command, ["--due"]) + + assert "Model emergent question" not in result.output + assert "1 open question(s) carry no cadence" in result.output + + def test_a_prediction_past_its_due_date_awaits_a_verdict(self, cli_runner, test_session): + q = make_question(test_session, "Does CMMC Phase 2 slip?", "90d") + # Staked legitimately, then time passed: due_by is set directly rather + # than through `predict`, which refuses a past date at entry. + p = Prediction( + question_id=q.id, + observable_text="Yes -- slips", + trigger_condition="Operator judgement.", + author=PREDICTION_AUTHOR_OPERATOR, + due_by=utcnow() - timedelta(days=3), + confidence=0.7, + status=PREDICTION_STATUS_OPEN, + ) + test_session.add(p) + test_session.commit() + + result = run(cli_runner, test_session, forecast_command, ["--due"]) + + assert "AWAITING YOUR VERDICT (1)" in result.output + assert "Yes -- slips" in result.output + assert f"resolve {p.id} --outcome yes|no" in result.output + + def test_a_prediction_not_yet_due_does_not_await_a_verdict(self, cli_runner, test_session): + q = make_question(test_session, "Does CMMC Phase 2 slip?", "90d") + run( + cli_runner, + test_session, + predict_command, + [str(q.id), "Yes -- slips", "--by", future_date(30), "--confidence", "0.7"], + ) + + result = run(cli_runner, test_session, forecast_command, ["--due"]) + + assert "AWAITING YOUR VERDICT (0)" in result.output + + +class TestAuthorSplitInTrackRecord: + @pytest.fixture + def mixed_ledger(self, test_session): + q = Question( + text="Will the rule slip?", + normalized_text="will the rule slip", + status=QUESTION_STATUS_OPEN, + cadence="30d", + ) + test_session.add(q) + test_session.flush() + run_row = AnalysisRun(run_type="situation_synthesis", status="completed") + test_session.add(run_row) + test_session.flush() + synth = NarrativeSynthesis(analysis_run_id=run_row.id) + test_session.add(synth) + test_session.flush() + + # The model's: a triggered and a contradicted observable. If these were + # counted, the operator's hit rate below would read 66%, not 50%. + for status in (PREDICTION_STATUS_TRIGGERED, PREDICTION_STATUS_CONTRADICTED): + test_session.add( + Prediction( + question_id=q.id, + observable_text=f"model observable {status}", + trigger_condition="coverage", + made_in_synthesis_id=synth.id, + author=PREDICTION_AUTHOR_MODEL, + status=status, + resolved_at=utcnow(), + ) + ) + # The operator's: one right, one wrong -> a 50% hit rate. + test_session.add( + Prediction( + question_id=q.id, + observable_text="operator claim that held", + trigger_condition="Operator judgement.", + author=PREDICTION_AUTHOR_OPERATOR, + due_by=utcnow() - timedelta(days=1), + confidence=0.8, + status=PREDICTION_STATUS_TRIGGERED, + outcome="yes", + resolved_at=utcnow(), + resolution_note="Happened.", + ) + ) + test_session.add( + Prediction( + question_id=q.id, + observable_text="operator claim that failed", + trigger_condition="Operator judgement.", + author=PREDICTION_AUTHOR_OPERATOR, + due_by=utcnow() - timedelta(days=1), + confidence=0.6, + status=PREDICTION_STATUS_CONTRADICTED, + outcome="no", + resolved_at=utcnow(), + resolution_note="Did not happen.", + ) + ) + test_session.commit() + return test_session + + def test_hit_rate_counts_operator_predictions_only(self, cli_runner, mixed_ledger): + result = run(cli_runner, mixed_ledger, predictions_command, ["track-record"]) + + assert result.exit_code == 0 + assert "YOUR CALIBRATION (operator predictions only)" in result.output + assert "Claims staked: 2" in result.output + assert "Hit rate: 1/2 = 50%" in result.output + + def test_model_predictions_are_reported_under_their_own_heading(self, cli_runner, mixed_ledger): + result = run(cli_runner, mixed_ledger, predictions_command, ["track-record"]) + + assert "MODEL PREDICTIONS (not counted toward your calibration)" in result.output + assert "2 made | 1 triggered | 1 contradicted" in result.output + + def test_the_two_are_never_mixed(self, cli_runner, mixed_ledger): + """ + Four predictions exist, two of each author. The calibration block must + never report four, and the model block must never report a hit rate. + """ + result = run(cli_runner, mixed_ledger, predictions_command, ["track-record"]) + + operator_block, model_block = result.output.split("MODEL PREDICTIONS") + assert "Claims staked: 4" not in operator_block + assert "Hit rate" in operator_block + assert "Hit rate" not in model_block + + def test_a_ledger_with_no_operator_claims_reports_no_figure(self, cli_runner, test_session): + q = Question(text="Q", normalized_text="q", status=QUESTION_STATUS_OPEN) + test_session.add(q) + test_session.flush() + test_session.add( + Prediction( + question_id=q.id, + observable_text="model observable", + trigger_condition="coverage", + author=PREDICTION_AUTHOR_MODEL, + status=PREDICTION_STATUS_TRIGGERED, + resolved_at=utcnow(), + ) + ) + test_session.commit() + + result = run(cli_runner, test_session, predictions_command, ["track-record"]) + + assert "You have staked nothing in this window" in result.output + assert "Hit rate" not in result.output + + +class TestNoApiKeyAndNoNetwork: + """ + Every command in the loop is local database work. Acceptance requires that + they work with no ANTHROPIC_API_KEY and make no network call, so the socket + is removed outright rather than the key merely unset. + """ + + @pytest.fixture + def no_network(self, monkeypatch): + monkeypatch.delenv("ANTHROPIC_API_KEY", raising=False) + + def _blocked(*args, **kwargs): + raise AssertionError("A calibration-loop command attempted a network call.") + + monkeypatch.setattr(socket, "socket", _blocked) + monkeypatch.setattr(socket, "create_connection", _blocked) + + def test_the_whole_loop_runs_offline(self, cli_runner, test_session, no_network): + added = run( + cli_runner, + test_session, + questions_command, + ["add", "Does the rule slip?", "--cadence", "7d"], + ) + assert added.exit_code == 0 + q = test_session.query(Question).one() + + staked = run( + cli_runner, + test_session, + predict_command, + [str(q.id), "Yes -- slips", "--by", future_date(20), "--confidence", "0.7"], + ) + assert staked.exit_code == 0 + p = test_session.query(Prediction).one() + + due = run(cli_runner, test_session, forecast_command, ["--due"]) + assert due.exit_code == 0 + + resolved = run( + cli_runner, + test_session, + resolve_command, + [str(p.id), "--outcome", "yes", "--note", "It slipped."], + ) + assert resolved.exit_code == 0 + + record = run(cli_runner, test_session, predictions_command, ["track-record"]) + assert record.exit_code == 0 + assert "Hit rate: 1/1 = 100%" in record.output diff --git a/tests/cli/test_ledger_scope.py b/tests/cli/test_ledger_scope.py index 699f025..98aa8a0 100644 --- a/tests/cli/test_ledger_scope.py +++ b/tests/cli/test_ledger_scope.py @@ -175,10 +175,13 @@ def test_track_record_counts_only_your_own_predictions(self, cli_runner, two_led """ The calibration number is the one figure that must never be polluted: four predictions exist, but only the two in this scope may be counted. + + The fixture's predictions are model-authored, so the count appears + under the model heading (backlog task 011 split the report by author). """ output = run(cli_runner, predictions_command, ["track-record"]) - assert "Predictions made: 2" in output + assert "2 made" in output class TestBeatScopeShowsOnlyTheBeat: @@ -204,7 +207,7 @@ def test_forecast_beat(self, cli_runner, two_ledgers): def test_track_record_beat(self, cli_runner, two_ledgers): output = run(cli_runner, predictions_command, ["track-record", "--beat", BEAT_NAME]) - assert "Predictions made: 2" in output + assert "2 made" in output class TestScopeIsVisibleInOutput: diff --git a/tests/cli/test_predictions_cli.py b/tests/cli/test_predictions_cli.py index 066a4ef..8b6c993 100644 --- a/tests/cli/test_predictions_cli.py +++ b/tests/cli/test_predictions_cli.py @@ -114,13 +114,20 @@ def test_lists_contradicted(self, cli_runner, populated_session): class TestTrackRecord: def test_track_record_summary(self, cli_runner, populated_session): + """ + The fixture's predictions are all model-authored, so they are reported + under the model heading and produce no hit rate. Backlog task 011 + moved the calibration figure to operator predictions only: a track + record that blends the two measures nothing. + """ session, _, _, _ = populated_session with _patch_db(session): result = cli_runner.invoke(predictions_command, ["track-record"]) assert result.exit_code == 0 - assert "Predictions made:" in result.output - # 1 triggered + 1 contradicted resolved -> 50% triggered. - assert "50%" in result.output + assert "MODEL PREDICTIONS (not counted toward your calibration)" in result.output + assert "3 made | 1 triggered | 1 contradicted | 1 open" in result.output + assert "You have staked nothing in this window" in result.output + assert "Hit rate" not in result.output def test_track_record_empty_window(self, cli_runner, populated_session): session, _, _, _ = populated_session diff --git a/tests/database/test_calibration_migration.py b/tests/database/test_calibration_migration.py new file mode 100644 index 0000000..b50df21 --- /dev/null +++ b/tests/database/test_calibration_migration.py @@ -0,0 +1,173 @@ +""" +Tests for the add_calibration_loop migration (backlog task 011). + +The migration rebuilds `predictions` to relax a NOT NULL constraint, which +SQLite cannot do in place. A rebuild that lost or misattributed a row would +corrupt the ledger this feature exists to make trustworthy, so it is exercised +against the real pre-011 DDL rather than against the current model. +""" + +import pytest +from sqlalchemy import create_engine, inspect, text + +from src.database.migrations import add_calibration_loop + +# The exact shape `predictions` and `questions` had before this migration, +# copied from a live database on 2026-08-27. +_PRE_011_SCHEMA = [ + """ + CREATE TABLE questions ( + id INTEGER NOT NULL, + text TEXT NOT NULL, + normalized_text TEXT NOT NULL, + first_asked_at DATETIME NOT NULL, + status VARCHAR(20) NOT NULL, + resolved_at DATETIME, + resolution_note TEXT, + previous_question_id INTEGER, + is_primary BOOLEAN NOT NULL, + PRIMARY KEY (id), + FOREIGN KEY(previous_question_id) REFERENCES questions (id) + ) + """, + "CREATE INDEX idx_question_status ON questions (status)", + """ + CREATE TABLE predictions ( + id INTEGER NOT NULL, + question_id INTEGER NOT NULL, + observable_text TEXT NOT NULL, + trigger_condition TEXT NOT NULL, + made_at DATETIME NOT NULL, + made_in_synthesis_id INTEGER NOT NULL, + status VARCHAR(20) NOT NULL, + resolved_at DATETIME, + resolution_note TEXT, + PRIMARY KEY (id), + FOREIGN KEY(question_id) REFERENCES questions (id) + ) + """, + "CREATE INDEX idx_prediction_status ON predictions (status)", + "CREATE INDEX idx_prediction_question ON predictions (question_id)", + "CREATE INDEX idx_prediction_made_at ON predictions (made_at)", +] + + +@pytest.fixture +def legacy_engine(tmp_path, monkeypatch): + """A database at the pre-011 shape, with the migration pointed at it.""" + engine = create_engine(f"sqlite:///{tmp_path / 'legacy.db'}") + with engine.begin() as conn: + for statement in _PRE_011_SCHEMA: + conn.execute(text(statement)) + conn.execute( + text( + "INSERT INTO questions (id, text, normalized_text, first_asked_at, " + "status, is_primary) VALUES (1, 'Will it slip?', 'will it slip', " + "'2026-01-01 00:00:00', 'open', 1)" + ) + ) + for i in range(1, 4): + conn.execute( + text( + "INSERT INTO predictions (id, question_id, observable_text, " + "trigger_condition, made_at, made_in_synthesis_id, status) VALUES " + f"({i}, 1, 'observable {i}', 'trigger {i}', '2026-02-0{i} 00:00:00', 7, 'open')" + ) + ) + monkeypatch.setattr(add_calibration_loop, "engine", engine) + return engine + + +def _columns(engine, table): + with engine.connect() as conn: + return {col["name"] for col in inspect(conn).get_columns(table)} + + +class TestUpgrade: + def test_adds_cadence_columns_to_questions(self, legacy_engine): + add_calibration_loop.upgrade() + + assert {"cadence", "last_reviewed_at"} <= _columns(legacy_engine, "questions") + + def test_adds_the_authorship_columns_to_predictions(self, legacy_engine): + add_calibration_loop.upgrade() + + assert {"author", "due_by", "confidence", "outcome"} <= _columns( + legacy_engine, "predictions" + ) + + def test_carries_every_existing_row_and_backfills_author_to_model(self, legacy_engine): + add_calibration_loop.upgrade() + + with legacy_engine.connect() as conn: + rows = conn.execute( + text("SELECT id, observable_text, author FROM predictions ORDER BY id") + ).all() + assert [r[0] for r in rows] == [1, 2, 3] + assert [r[1] for r in rows] == ["observable 1", "observable 2", "observable 3"] + assert {r[2] for r in rows} == {"model"} + + def test_made_in_synthesis_id_becomes_nullable(self, legacy_engine): + """The whole reason for the rebuild: an operator claim has no synthesis.""" + add_calibration_loop.upgrade() + + with legacy_engine.begin() as conn: + conn.execute( + text( + "INSERT INTO predictions (question_id, observable_text, " + "trigger_condition, made_at, status, author) VALUES " + "(1, 'Yes -- slips', 'Operator judgement.', " + "'2026-08-27 00:00:00', 'open', 'operator')" + ) + ) + stored = conn.execute( + text("SELECT made_in_synthesis_id FROM predictions WHERE author = 'operator'") + ).scalar_one() + assert stored is None + + def test_is_idempotent(self, legacy_engine): + add_calibration_loop.upgrade() + add_calibration_loop.upgrade() + + with legacy_engine.connect() as conn: + assert conn.execute(text("SELECT COUNT(*) FROM predictions")).scalar_one() == 3 + + def test_leaves_no_scratch_table_behind(self, legacy_engine): + add_calibration_loop.upgrade() + + with legacy_engine.connect() as conn: + names = set(inspect(conn).get_table_names()) + assert "predictions_pre_011" not in names + + +class TestDowngrade: + def test_restores_the_pre_011_shape(self, legacy_engine): + add_calibration_loop.upgrade() + add_calibration_loop.downgrade() + + assert {"author", "due_by", "confidence", "outcome"}.isdisjoint( + _columns(legacy_engine, "predictions") + ) + assert {"cadence", "last_reviewed_at"}.isdisjoint(_columns(legacy_engine, "questions")) + with legacy_engine.connect() as conn: + assert conn.execute(text("SELECT COUNT(*) FROM predictions")).scalar_one() == 3 + + def test_refuses_rather_than_misattributing_an_operator_claim(self, legacy_engine): + add_calibration_loop.upgrade() + with legacy_engine.begin() as conn: + conn.execute( + text( + "INSERT INTO predictions (question_id, observable_text, " + "trigger_condition, made_at, status, author) VALUES " + "(1, 'Yes -- slips', 'Operator judgement.', " + "'2026-08-27 00:00:00', 'open', 'operator')" + ) + ) + + with pytest.raises(RuntimeError, match="operator prediction"): + add_calibration_loop.downgrade() + + # Refusing must leave the ledger untouched, not half-reverted. + assert "author" in _columns(legacy_engine, "predictions") + with legacy_engine.connect() as conn: + assert conn.execute(text("SELECT COUNT(*) FROM predictions")).scalar_one() == 4 diff --git a/tests/utils/test_cadence.py b/tests/utils/test_cadence.py new file mode 100644 index 0000000..6d45a21 --- /dev/null +++ b/tests/utils/test_cadence.py @@ -0,0 +1,88 @@ +""" +Tests for review-cadence parsing (backlog task 011). + +A cadence is not a deadline: it says how often a question is worth +re-examining. These tests pin the interval arithmetic that `forecast --due` +depends on to surface two questions on different cadences independently. +""" + +from datetime import datetime, timedelta + +import pytest + +from src.utils.cadence import ( + InvalidCadence, + describe_next_review, + is_due, + next_review_at, + normalize_cadence, + parse_cadence, +) + +NOW = datetime(2026, 8, 27, 12, 0, 0) + + +class TestParse: + @pytest.mark.parametrize( + ("raw", "days"), + [("7d", 7), ("30d", 30), ("90d", 90), ("2w", 14), ("3m", 90), ("1y", 365)], + ) + def test_reads_the_forms_the_help_text_advertises(self, raw, days): + assert parse_cadence(raw) == timedelta(days=days) + + @pytest.mark.parametrize("raw", [" 7d ", "7D", "1Y"]) + def test_tolerates_case_and_whitespace(self, raw): + assert parse_cadence(raw) > timedelta(0) + + @pytest.mark.parametrize("raw", ["", None, "soon", "d", "7", "7x", "-3d", "0d", "1.5y"]) + def test_refuses_anything_it_cannot_read(self, raw): + with pytest.raises(InvalidCadence): + parse_cadence(raw) + + def test_normalizes_to_a_canonical_spelling(self): + assert normalize_cadence(" 90D ") == "90d" + + +class TestNextReview: + def test_counts_from_first_asked_when_never_reviewed(self): + first_asked = NOW - timedelta(days=3) + assert next_review_at("7d", None, first_asked) == first_asked + timedelta(days=7) + + def test_counts_from_the_last_review_once_reviewed(self): + first_asked = NOW - timedelta(days=300) + reviewed = NOW - timedelta(days=2) + assert next_review_at("7d", reviewed, first_asked) == reviewed + timedelta(days=7) + + def test_no_cadence_means_no_next_review(self): + assert next_review_at(None, None, NOW) is None + + +class TestIsDue: + def test_not_due_before_the_interval_elapses(self): + assert not is_due("90d", None, NOW - timedelta(days=10), NOW) + + def test_due_once_the_interval_elapses(self): + assert is_due("7d", None, NOW - timedelta(days=10), NOW) + + def test_two_cadences_are_independent_at_the_same_instant(self): + first_asked = NOW - timedelta(days=10) + assert is_due("7d", None, first_asked, NOW) + assert not is_due("90d", None, first_asked, NOW) + + def test_a_question_with_no_cadence_is_never_due(self): + assert not is_due(None, None, NOW - timedelta(days=4000), NOW) + + +class TestDescribe: + def test_says_how_long_until_the_next_review(self): + assert describe_next_review("90d", None, NOW - timedelta(days=10), NOW) == ( + "next review in 80d" + ) + + def test_says_how_overdue_it_is(self): + assert describe_next_review("7d", None, NOW - timedelta(days=10), NOW) == ( + "due now (3d overdue)" + ) + + def test_says_no_cadence_rather_than_inventing_one(self): + assert describe_next_review(None, None, NOW, NOW) == "no cadence" From 6a82403d6831bd7706b5e6f94489d741254d20fd Mon Sep 17 00:00:00 2001 From: sam-aydlette Date: Thu, 27 Aug 2026 14:05:40 -0400 Subject: [PATCH 2/4] beats: test a beat against events that actually happened, not article counts Task 005 asked whether a beat could reach its domain and answered with document counts -- 469 Federal Register documents narrowed to 24, 38 in seven days. Both real, both green, and neither able to detect that the beat was structurally unable to see a personnel change, because a personnel change is not a document. The first live brief then missed the reinstatement of the FedRAMP director. The corpus held 3 incidental FedRAMP mentions across 50,983 articles and none within two weeks. An article count measures whether ingestion is running. It does not measure whether ingestion reaches the domain. `insightweaver beat coverage NAME` measures the second thing: a beat declares known past events as `coverage_probes`, and the command reports per probe whether a feed the beat subscribes to carried a matching article, and which one. - `terms` must all appear; each `any_of` group needs one member. A probe resting on a single bare term is rejected by the loader -- `FedRAMP` alone is matched by an AWS region-launch post, and a probe that passes on generic terms manufactures the confidence this exists to remove. - Matching is deterministic word-boundary matching through the same primitives coverage entities use. No model call. Measured on the corpus: 1,842 articles contain the substring "mail", 339 contain it as a word; 5,364 contain "nist" against 73 as a word. - A trailing `*` marks a stem (`reinstat*`). Explicit rather than inferred, so the widening is visible to whoever has to trust the result. - A probe whose window holds no articles from the beat's sources is INCONCLUSIVE, with the reason distinguishing "predates the corpus" from "corpus has not reached it" from a gap. It stays in the denominator; a probe set that decays to the events still in retention is a green light that means nothing. - An unmatched probe is re-checked corpus-wide, so "nobody carried it" and "feeds you do not subscribe to carried it" are distinguishable. - Exits 1 on any unmatched probe, 2 when nothing was measured (no probes declared, or all inconclusive). Reads only; never writes. The shipped beat declares four probes. Against the current corpus two match, the FedRAMP reinstatement does not, and BOD 22-01 predates the corpus -- which is the honest measurement task 010 was written to get. --- README.md | 69 +++ config/beats/us-public-sector-compliance.json | 35 ++ src/cli/app.py | 11 +- src/cli/beat.py | 221 ++++++++ src/config/beats.py | 205 +++++++- src/context/coverage_probe.py | 488 ++++++++++++++++++ src/context/entity_matcher.py | 26 +- tests/cli/test_beat_coverage.py | 304 +++++++++++ tests/config/test_beat_probes.py | 194 +++++++ tests/context/test_coverage_probe.py | 403 +++++++++++++++ 10 files changed, 1948 insertions(+), 8 deletions(-) create mode 100644 src/cli/beat.py create mode 100644 src/context/coverage_probe.py create mode 100644 tests/cli/test_beat_coverage.py create mode 100644 tests/config/test_beat_probes.py create mode 100644 tests/context/test_coverage_probe.py diff --git a/README.md b/README.md index 17a7c4c..5568190 100644 --- a/README.md +++ b/README.md @@ -318,6 +318,75 @@ reads as inactivity. A named individual may appear inside a rendered situation w source document names a signatory -- an attribute of a document, which expires with it -- but never as a stored row, which would accumulate. See `docs/CONCEPTS.md`. +### Coverage probes: can the beat actually see its domain? + +Added 2026-08-27. An article count tells you ingestion is *running*. It does not tell +you ingestion *reaches the domain*, and the two were conflated once already: a beat +resolving hundreds of Federal Register documents missed the reinstatement of the +FedRAMP director, because a personnel change is not a document. The corpus held three +incidental FedRAMP mentions across 50,983 articles and none within two weeks. Every +volume number was green. + +A **coverage probe** is the test that catches it: name something that actually happened +in this domain, then check whether the beat can see it. + +```json +"coverage_probes": [ + { + "date": "2026-08-24", + "what": "FedRAMP director reinstated", + "terms": ["FedRAMP"], + "any_of": [["director", "administrator"], ["reinstat*", "return*", "restored"]] + } +] +``` + +```bash +insightweaver beat coverage us-public-sector-compliance +``` + +``` +[MATCHED] OPM issued corrections to its final rules on reduction-in-force appeals + event 2026-08-25 | window 2026-08-11 .. 2026-09-08 + 4 matching article(s); earliest shown + seen by Federal Register - Documents API (2026-08-25) + Suitability Action Appeals; Correction + +[UNMATCHED] FedRAMP director reinstated + no article from this beat's feeds matched + closest article lacked: FedRAMP + and no feed anywhere in the corpus carried it either. + +2 matched + 1 unmatched + 1 inconclusive of 4 probe(s) declared +``` + +`terms` must all appear in one article; each `any_of` group needs one member. The two +levels exist because `FedRAMP` alone is too weak to be evidence -- an AWS region-launch +post mentions it -- while an exact headline is too brittle to survive a second outlet's +phrasing. A probe resting on one bare term is **rejected by the loader**, not accepted: +a probe that passes on generic terms manufactures the confidence this command exists to +remove. + +Matching is deterministic word-boundary matching, no model call. That anchoring is what +makes a match evidence: of 54,044 articles stored on 2026-08-27, 657 titles contain the +substring `nist` -- "administration", "minister", "Afghanistan", "communist" -- and four +are about NIST. A trailing `*` marks a stem (`reinstat*` matches "reinstatement"); the +marker is explicit rather than inferred so the widening is visible to whoever has to +trust the result. + +The command reports **which feed carried each match**, because "your expected source +carried it" and "an unrelated outlet mentioned it in passing" are different findings +with different repairs. An unmatched probe is re-checked across the whole corpus, so the +output distinguishes "nobody had it" from "feeds you do not subscribe to had it". + +It exits non-zero so it can gate: `1` if any probe is unmatched, `2` if nothing could be +measured. A probe whose window predates the corpus is `INCONCLUSIVE` -- neither pass nor +fail -- and stays in the denominator, because a probe set that quietly decays to the +events still in retention is a green light that means nothing. That is also why an +all-inconclusive run and a beat with no probes both exit `2` rather than `0`. + +Reads the article corpus only. No API key, no network, and it never writes. + Running your first beat needs the two beat tables, plus one each for institutional activity and standing questions: diff --git a/config/beats/us-public-sector-compliance.json b/config/beats/us-public-sector-compliance.json index 9eadfed..ef6193a 100644 --- a/config/beats/us-public-sector-compliance.json +++ b/config/beats/us-public-sector-compliance.json @@ -49,5 +49,40 @@ "Does any CISA BOD create a compliance obligation with a deadline inside 90 days?", "Where do GovRAMP and TX-RAMP diverge from FedRAMP in ways that matter to a multi-state CSP?" ], + "coverage_probes": [ + { + "date": "2026-08-25", + "what": "OPM issued corrections to its final rules on reduction-in-force, suitability and probationary appeals", + "terms": ["OPM"], + "any_of": [ + ["reduction in force", "suitability", "probationary"], + ["final rule", "correction"] + ] + }, + { + "date": "2026-08-24", + "what": "Supreme Court let the executive order restricting mail-in voting take effect", + "terms": ["Supreme Court", "mail-in voting"], + "any_of": [["allow*", "restriction*", "sides with", "stay"]] + }, + { + "date": "2026-08-24", + "what": "FedRAMP director reinstated", + "terms": ["FedRAMP"], + "any_of": [ + ["director", "administrator"], + ["reinstat*", "return*", "restored"] + ] + }, + { + "date": "2021-11-03", + "what": "CISA issued Binding Operational Directive 22-01, creating the Known Exploited Vulnerabilities catalog and its remediation deadlines", + "terms": ["CISA"], + "any_of": [ + ["binding operational directive", "BOD 22-01"], + ["known exploited", "catalog"] + ] + } + ], "channels": ["terminal"] } diff --git a/src/cli/app.py b/src/cli/app.py index dad6945..2674748 100644 --- a/src/cli/app.py +++ b/src/cli/app.py @@ -9,6 +9,7 @@ import click +from .beat import beat_command from .brief import brief_group from .colors import accent, header, muted from .decisions import decisions_command @@ -26,6 +27,7 @@ # In particular "predictions" must precede "predict", since the shorter name is # a prefix of the longer one (2026-08-27, backlog task 011). COMMAND_DISPATCH = { + "beat": beat_command, "brief": brief_group, "forecast": forecast_command, "frames": frames_command, @@ -42,7 +44,7 @@ def print_command_refresher(): """Print a short refresher of available commands.""" refresher = ( - f"\n{header('Commands:')} {accent('brief')} | {accent('forecast')} | " + f"\n{header('Commands:')} {accent('brief')} | {accent('beat')} | {accent('forecast')} | " f"{accent('frames')} | {accent('questions')} | {accent('predictions')} | " f"{accent('predict')} | {accent('resolve')} | " f"{accent('decisions')} | {accent('diet')} | {accent('sources')} | " @@ -55,6 +57,7 @@ def print_help(): """Print full help text for interactive mode.""" click.echo(header("Available commands:")) click.echo(f" {accent('brief')} - Generate intelligence brief and report") + click.echo(f" {accent('beat')} - Test a beat's sources against real events") click.echo( f" {accent('forecast')} - View open observables and resolved track record" ) @@ -128,6 +131,11 @@ def print_help(): click.echo(f" {accent('diet gaps')} - Recurring frame absences") click.echo(f" {accent('diet overlap')} - Which frames only one feed carries") click.echo() + click.echo(header("Beat command:")) + click.echo(f" {accent('beat coverage')} - Can this beat see events that happened?") + click.echo(muted(" Reads config/beats/NAME.json coverage_probes. No API key, no writes.")) + click.echo(muted(" Exits 1 if a probe is unmatched, 2 if nothing could be measured.")) + click.echo() click.echo(header("Sources command:")) click.echo(f" {accent('sources list')} - All feeds with calibration signals") click.echo(f" {accent('sources show')} - Detailed view for one feed") @@ -139,6 +147,7 @@ def print_help(): click.echo(muted(" brief --save brief.md (save brief as markdown)")) click.echo(muted(" brief --from-run 176 (re-render stored brief 176)")) click.echo(muted(" brief --beat us-public-sector-compliance (subject brief)")) + click.echo(muted(" beat coverage us-public-sector-compliance (can it see real events?)")) click.echo(muted(" forecast (open observables + last 60d resolved)")) click.echo(muted(" forecast --days 30 (tighter resolved-record window)")) click.echo(muted(" forecast --due (what is due now, at each question's cadence)")) diff --git a/src/cli/beat.py b/src/cli/beat.py new file mode 100644 index 0000000..021b210 --- /dev/null +++ b/src/cli/beat.py @@ -0,0 +1,221 @@ +""" +Beat Command - test a beat's sources against events that actually happened. + +``insightweaver beat coverage NAME`` runs the beat's declared +``coverage_probes`` against the stored corpus and reports, per probe, whether an +article the beat's own sources carried would have shown the operator the event, +and which source that was. + +It exits non-zero when a probe went unmatched, so it can gate a release the way +a test does. It also exits non-zero when *nothing was measured* -- no probes +declared, or every probe inconclusive -- because "no evidence of a gap" and "no +evidence" are the same green tick and only one of them is good news. That +conflation is the whole reason backlog task 010 exists. + +No model is called and nothing is written. Added 2026-08-27. +""" + +import click + +from ..config.beats import BeatNotFound, BeatValidationError, load_beat +from ..context.coverage_probe import ( + STATUS_INCONCLUSIVE, + STATUS_MATCHED, + STATUS_UNMATCHED, + CoverageReport, + ProbeResult, + run_coverage_probes, +) +from ..database.connection import SessionLocal +from .colors import accent, error, header, muted, success, warning + +# Exit codes. Distinct on purpose: a gate that fails and a gate that could not +# run are different alarms, and collapsing them would let a rotted probe set +# read as a broken beat (or, worse, the reverse). +EXIT_OK = 0 +EXIT_UNMATCHED = 1 +EXIT_NOTHING_MEASURED = 2 + +_STATUS_STYLE = { + STATUS_MATCHED: success, + STATUS_UNMATCHED: error, + STATUS_INCONCLUSIVE: warning, +} + + +@click.group(name="beat") +def beat_command(): + """Inspect a beat's configuration against the corpus.""" + pass + + +@beat_command.command(name="coverage") +@click.argument("name") +@click.option( + "--show", + type=click.Choice(["all", "problems"]), + default="all", + help="'problems' prints only the probes that did not pass.", +) +def coverage(name, show): + """ + Check whether this beat's sources can see events that actually happened. + + Each probe in the beat's `coverage_probes` names a real past event and the + words any report of it would carry. A probe passes only when a feed this + beat subscribes to carried a matching article inside the probe's window. + + Exits 1 if any probe is unmatched, 2 if nothing could be measured. + """ + try: + beat = load_beat(name) + except (BeatNotFound, BeatValidationError) as exc: + raise click.ClickException(str(exc)) + + if not beat.coverage_probes: + click.echo(error(f"Beat '{beat.name}' declares no coverage_probes.")) + click.echo( + muted( + "Nothing was measured. An article count tells you ingestion is running;\n" + "only a probe tells you it reaches this domain. Name three things that\n" + "actually happened in this beat's subject this month and declare them in\n" + f"{beat.config_path} under 'coverage_probes'." + ) + ) + raise SystemExit(EXIT_NOTHING_MEASURED) + + # Read-only by construction: this session is never committed, because the + # corpus is shared and a question about it must not alter it. + session = SessionLocal() + try: + report = run_coverage_probes(session, beat) + finally: + session.rollback() + session.close() + + _print_report(report, beat.config_path, show) + code = _exit_code(report) + if code != EXIT_OK: + raise SystemExit(code) + + +def _exit_code(report: CoverageReport) -> int: + """ + The verdict, as a shell sees it. + + ``EXIT_NOTHING_MEASURED`` for an all-inconclusive run is the load-bearing + case: with no probe answered there is no evidence the beat reaches anything, + and exiting 0 there would hand a gate the same green tick a real pass gets. + """ + if report.unmatched: + return EXIT_UNMATCHED + if not report.conclusive: + return EXIT_NOTHING_MEASURED + return EXIT_OK + + +def _print_report(report: CoverageReport, config_path: str, show: str) -> None: + click.echo(header(f"COVERAGE PROBES - beat '{report.beat_name}'")) + click.echo("=" * 80) + span = ( + f"{report.earliest.date().isoformat()} to {report.latest.date().isoformat()}" + if report.earliest and report.latest + else "no dated articles" + ) + click.echo(muted(f" sources: {len(report.feed_names)} feeds | corpus span: {span}")) + click.echo(muted(f" probes declared in {config_path}")) + click.echo() + + for result in report.results: + if show == "problems" and result.matched: + continue + _print_result(result) + + _print_summary(report) + + +def _print_result(result: ProbeResult) -> None: + style = _STATUS_STYLE[result.status] + start, end = result.window + click.echo(f"{style(f'[{result.status}]')} {accent(result.probe.what)}") + click.echo(muted(f" event {result.probe.date.isoformat()} | window {start} .. {end}")) + click.echo(muted(f" requires {result.probe.describe()}")) + click.echo(muted(f" {result.articles_in_window} article(s) from this beat's feeds in window")) + + if result.status == STATUS_MATCHED: + click.echo(muted(f" {result.match_count} matching article(s); earliest shown")) + for match in result.matches: + click.echo( + f" {success('seen by')} {accent(match.feed_name)} " + f"{muted(f'({match.published_day})')}" + ) + click.echo(muted(f" {match.title}")) + elif result.status == STATUS_UNMATCHED: + click.echo(error(" no article from this beat's feeds matched")) + if result.missing_terms: + click.echo(muted(f" closest article lacked: {', '.join(result.missing_terms)}")) + for group in result.unsatisfied_groups: + click.echo(muted(f" closest article lacked any of: {', '.join(group)}")) + if result.elsewhere: + click.echo( + warning( + f" but {result.elsewhere_count} article(s) elsewhere in the corpus " + f"matched, outside this beat's sources:" + ) + ) + for match in result.elsewhere: + click.echo(f" {accent(match.feed_name)} {muted(f'({match.published_day})')}") + click.echo(muted(f" {match.title}")) + click.echo(muted(" the sources are reachable; this beat's list is what missed.")) + else: + click.echo(muted(" and no feed anywhere in the corpus carried it either.")) + else: + click.echo(warning(f" {result.reason}")) + click.echo(muted(" neither a pass nor a failure; still counted below.")) + click.echo() + + +def _print_summary(report: CoverageReport) -> None: + """ + The counts, with the denominator always the number of probes declared. + + Printed as `matched + unmatched + inconclusive of N declared` so the + arithmetic is visible. A probe set decaying out of retention has to show up + here as a shrinking number of *answered* probes rather than as a clean sweep + of the few that still match. + """ + click.echo("=" * 80) + matched, unmatched, inconclusive = ( + len(report.matched), + len(report.unmatched), + len(report.inconclusive), + ) + click.echo( + f"{success(f'{matched} matched')} + {error(f'{unmatched} unmatched')} + " + f"{warning(f'{inconclusive} inconclusive')} of {report.total} probe(s) declared" + ) + + if unmatched: + click.echo( + error( + f"FAIL: {unmatched} event(s) this beat claims to cover left no trace " + f"in its sources." + ) + ) + elif not report.conclusive: + click.echo( + warning( + "NOT MEASURED: every probe was inconclusive. This is not a pass -- the " + "corpus could not answer a single one of them." + ) + ) + else: + click.echo(success(f"PASS: {matched} of {report.total} probe(s) answered, none missed.")) + + if inconclusive and not unmatched and report.conclusive: + click.echo( + muted( + f" {inconclusive} probe(s) could not be answered and are counted above, " + f"not dropped." + ) + ) diff --git a/src/config/beats.py b/src/config/beats.py index 8f67613..d5a22d4 100644 --- a/src/config/beats.py +++ b/src/config/beats.py @@ -20,6 +20,7 @@ import json from dataclasses import dataclass +from datetime import date, timedelta from pathlib import Path from typing import Any @@ -41,12 +42,39 @@ SUPPORTED_CHANNELS = frozenset({"terminal", "markdown", "html", "email"}) BEAT_KEYS = frozenset( - {"name", "description", "sources", "coverage", "standing_questions", "channels"} + { + "name", + "description", + "sources", + "coverage", + "standing_questions", + "coverage_probes", + "channels", + } ) REQUIRED_BEAT_KEYS = frozenset({"name", "sources"}) SOURCE_KEYS = frozenset({"adapter", "feed_tags", "geo_tags", "scope"}) REQUIRED_SOURCE_KEYS = frozenset({"adapter", "feed_tags"}) +# Keys permitted on one coverage probe (backlog task 010, 2026-08-27). +PROBE_KEYS = frozenset({"date", "what", "terms", "any_of", "window_days"}) +REQUIRED_PROBE_KEYS = frozenset({"date", "what", "terms"}) + +# How far either side of a probe's date a report of it is still a report of it. +# Symmetric because coverage runs both ways: a rule is trailed before it lands +# and analysed for a week after. Fourteen days is the interval the FedRAMP miss +# was measured over -- "3 incidental mentions and none within two weeks". +DEFAULT_PROBE_WINDOW_DAYS = 14 + +# The least evidence a probe may rest on. A probe is a claim that a match means +# the beat saw a specific event, and one bare term cannot carry that claim: +# `FedRAMP` alone is matched by an AWS region-launch post. Two independent +# pieces of evidence -- two required terms, or one required term plus one +# `any_of` group -- is the floor, enforced here rather than left to the author's +# judgement, because a probe that passes on generic terms manufactures exactly +# the confidence this feature exists to remove. +MIN_PROBE_EVIDENCE = 2 + # The only three things a beat may track, and the plural config key that # declares each. The mapping is closed on purpose: any other key -- `people`, # `officials`, `staff`, anything -- is a validation error rather than a @@ -152,6 +180,52 @@ def terms(self) -> tuple[str, ...]: return tuple(seen) +@dataclass(frozen=True) +class CoverageProbe: + """ + One thing that actually happened, used to test whether the beat can see it. + + A probe is not a search. It is a claim of the form "an event of this + description occurred on this date, and any report of it would carry these + words" -- which makes an unmatched probe a statement about the beat's + sources rather than about the phrasing of a query. + + ``terms`` must **all** appear in the same article; each group in ``any_of`` + contributes one alternative that must appear. The two levels exist because + a single distinctive term is too weak to be evidence and a whole headline is + too brittle to survive a second outlet's phrasing. + + A term ending in ``*`` is a stem: ``reinstat*`` matches "reinstated" and + "reinstatement". Without the marker a term matches whole words only. The + marker is explicit rather than implied so that the widening is visible in + the config to whoever has to trust the result -- see + :mod:`src.context.coverage_probe` for the matching rules themselves. + """ + + date: date + what: str + terms: tuple[str, ...] + any_of: tuple[tuple[str, ...], ...] = () + window_days: int = DEFAULT_PROBE_WINDOW_DAYS + + @property + def window(self) -> tuple[date, date]: + """The inclusive date range in which a report of this event counts.""" + span = timedelta(days=self.window_days) + return self.date - span, self.date + span + + @property + def evidence_count(self) -> int: + """How many independent things this probe requires of an article.""" + return len(self.terms) + len(self.any_of) + + def describe(self) -> str: + """The probe's requirement, written the way the config wrote it.""" + parts = [" AND ".join(self.terms)] if self.terms else [] + parts.extend("(" + " OR ".join(group) + ")" for group in self.any_of) + return " AND ".join(parts) + + @dataclass(frozen=True) class BeatConfig: """ @@ -164,6 +238,10 @@ class BeatConfig: ``standing_questions`` are the questions this beat declares it is watching, whether or not any given run's coverage mentions them. They are read by ``src/context/standing_agenda.py`` (backlog task 007). + + ``coverage_probes`` are known past events the beat is tested against by + ``insightweaver beat coverage`` (backlog task 010). They are the answer to + "can this beat reach its domain" that an article count cannot give. """ name: str @@ -174,6 +252,7 @@ class BeatConfig: channels: tuple[str, ...] config_path: str entities: tuple[CoverageEntity, ...] = () + coverage_probes: tuple[CoverageProbe, ...] = () def resolve_feeds(self, matcher: FeedMatcher | None = None) -> list[Feed]: """ @@ -255,6 +334,8 @@ def load_beat(name: str, beats_dir: Path | str | None = None) -> BeatConfig: standing_questions = _parse_standing_questions(path, raw.get("standing_questions", [])) + coverage_probes = _parse_coverage_probes(path, raw.get("coverage_probes", [])) + channels = _parse_channels(path, raw.get("channels", ["terminal"])) return BeatConfig( @@ -266,9 +347,131 @@ def load_beat(name: str, beats_dir: Path | str | None = None) -> BeatConfig: channels=channels, config_path=str(path), entities=entities, + coverage_probes=coverage_probes, ) +def _parse_coverage_probes(path: Path, raw_probes: Any) -> tuple[CoverageProbe, ...]: + """ + Validate the ``coverage_probes`` block into :class:`CoverageProbe` values. + + Every rule here refuses a probe rather than repairing it. A probe is + evidence about whether the beat's sources reach its domain, and a probe + that was quietly loosened -- an unparseable date defaulted to today, a + single generic term accepted -- would answer the question with something + other than what the author wrote. + """ + if not isinstance(raw_probes, list): + raise BeatValidationError(path, "'coverage_probes' must be a list of objects") + + parsed: list[CoverageProbe] = [] + for index, entry in enumerate(raw_probes): + where = f"coverage_probes[{index}]" + if not isinstance(entry, dict): + raise BeatValidationError(path, f"{where} must be an object, got {entry!r}") + + unknown = set(entry) - PROBE_KEYS + if unknown: + supported = ", ".join(sorted(PROBE_KEYS)) + raise BeatValidationError( + path, + f"{where} has unknown key(s): {', '.join(sorted(unknown))} (supported: {supported})", + ) + missing = REQUIRED_PROBE_KEYS - set(entry) + if missing: + raise BeatValidationError( + path, f"{where} missing required key(s): {', '.join(sorted(missing))}" + ) + + raw_date = entry["date"] + if not isinstance(raw_date, str): + raise BeatValidationError(path, f"{where}.date must be a 'YYYY-MM-DD' string") + try: + when = date.fromisoformat(raw_date) + except ValueError: + raise BeatValidationError( + path, f"{where}.date is not a 'YYYY-MM-DD' date: {raw_date!r}" + ) + + what = entry["what"] + if not isinstance(what, str) or not what.strip(): + raise BeatValidationError( + path, f"{where}.what must be a non-empty description of the event" + ) + + terms = _parse_probe_terms(path, f"{where}.terms", entry["terms"]) + if not terms: + raise BeatValidationError(path, f"{where}.terms must not be empty") + + raw_any_of = entry.get("any_of", []) + if not isinstance(raw_any_of, list): + raise BeatValidationError(path, f"{where}.any_of must be a list of term lists") + any_of: list[tuple[str, ...]] = [] + for group_index, group in enumerate(raw_any_of): + alternatives = _parse_probe_terms( + path, f"{where}.any_of[{group_index}]", group, allow_bare_string=False + ) + if not alternatives: + raise BeatValidationError( + path, + f"{where}.any_of[{group_index}] must not be empty -- an empty group of " + f"alternatives is satisfied by nothing and would make the probe unmatchable", + ) + any_of.append(alternatives) + + window_days = entry.get("window_days", DEFAULT_PROBE_WINDOW_DAYS) + if not isinstance(window_days, int) or isinstance(window_days, bool) or window_days < 1: + raise BeatValidationError( + path, f"{where}.window_days must be a positive integer, got {window_days!r}" + ) + + probe = CoverageProbe( + date=when, + what=what.strip(), + terms=terms, + any_of=tuple(any_of), + window_days=window_days, + ) + if probe.evidence_count < MIN_PROBE_EVIDENCE: + raise BeatValidationError( + path, + f"{where} rests on a single term ({terms[0]!r}). A probe needs at least " + f"{MIN_PROBE_EVIDENCE} independent requirements -- another entry in 'terms', " + f"or an 'any_of' group -- because one term on its own is matched by articles " + f"that merely mention the subject, and a probe that passes on a passing " + f"mention is worse than no probe.", + ) + parsed.append(probe) + + return tuple(parsed) + + +def _parse_probe_terms( + path: Path, where: str, value: Any, allow_bare_string: bool = True +) -> tuple[str, ...]: + """Validate one list of probe terms, order-preserving and deduplicated.""" + if allow_bare_string and isinstance(value, str): + value = [value] + if not isinstance(value, list): + raise BeatValidationError(path, f"{where} must be a list of strings") + + seen: dict[str, None] = {} + for term in value: + if not isinstance(term, str) or not term.strip(): + raise BeatValidationError( + path, f"{where} must contain only non-empty strings, got {term!r}" + ) + stripped = term.strip() + if stripped.rstrip("*") == "": + raise BeatValidationError( + path, + f"{where} contains {term!r}, which is a stem marker with no stem and " + f"would match every article", + ) + seen.setdefault(stripped, None) + return tuple(seen) + + def _parse_coverage(path: Path, coverage: dict[str, Any]) -> tuple[CoverageEntity, ...]: """ Validate the ``coverage`` block into :class:`CoverageEntity` values. diff --git a/src/context/coverage_probe.py b/src/context/coverage_probe.py new file mode 100644 index 0000000..8758ea1 --- /dev/null +++ b/src/context/coverage_probe.py @@ -0,0 +1,488 @@ +""" +Testing a beat against events that actually happened. + +Task 005 asked whether a beat could reach its domain and answered with document +counts: 469 Federal Register documents narrowed to 24, 38 retrieved in seven +days. Both real, both green, and neither able to detect that the beat was +structurally unable to see a personnel change -- because a personnel change is +not a document. The beat's first live brief then missed the reinstatement of the +FedRAMP director, the largest event in its declared domain that week. + +**An article count measures whether ingestion is running. It does not measure +whether ingestion reaches the domain.** A coverage probe measures the second +thing, by the only method that cannot be satisfied by ingesting more of what the +beat already reads: + + Name something that actually happened in this domain, then check whether the + beat can see it. + +Nothing here calls a model. A probe either matched a stored article or it did +not, and the answer is reproducible from the same rows forever and debuggable by +reading a regex rather than by re-running a prompt. + +Three rules do the work: + +* **Word boundaries, not substrings.** Measured on this repository's corpus on + 2026-08-27: of 54,044 stored articles, 657 titles contain the substring + ``nist`` -- "administration", "minister", "Afghanistan", "communist", + "sinister", "columnist" -- and exactly 4 are about NIST. A substring matcher + would report 99.4% false coverage on that one term alone. Terms are anchored + at word boundaries by :mod:`src.context.entity_matcher`, which is the same + code path coverage entities use. +* **Stems are marked, never inferred.** ``reinstat*`` matches "reinstated" and + "reinstatement"; ``reinstat`` matches neither. Inferring the widening would + make every term quietly broader than it reads in the config, which is the + direction that manufactures false confidence. +* **A window with no articles in it is not a failure.** A probe whose window + predates the corpus, or falls in a gap, is ``INCONCLUSIVE`` -- it says nothing + about the beat's reach, because there was nothing to reach with. It stays in + the count regardless, since a probe set that quietly shrinks to the events + still in retention is a green light that means nothing. + +Added 2026-08-27 for backlog task 010. +""" + +from __future__ import annotations + +import re +from collections.abc import Iterable, Sequence +from dataclasses import dataclass +from datetime import date, datetime, time + +from sqlalchemy.orm import Session + +from ..config.beats import BeatConfig, CoverageProbe +from ..database.models import Article, RSSFeed +from .entity_matcher import is_shouted, term_pattern + +__all__ = [ + "STATUS_INCONCLUSIVE", + "STATUS_MATCHED", + "STATUS_UNMATCHED", + "CompiledProbe", + "CoverageReport", + "ProbeMatch", + "ProbeResult", + "compile_probe", + "run_coverage_probes", +] + +# A probe matched at least one article the beat's own sources carried. +STATUS_MATCHED = "MATCHED" +# The window held articles from the beat's sources and none of them matched. +# This is the finding the feature exists to produce. +STATUS_UNMATCHED = "UNMATCHED" +# The window held nothing to match against, so the probe answers nothing. +STATUS_INCONCLUSIVE = "INCONCLUSIVE" + +# The marker that turns a term into a stem. Trailing only: a leading wildcard +# would drop the left-hand word boundary, which is the boundary that does +# essentially all of the collision filtering. +STEM_MARKER = "*" + +# How many matching articles to keep per probe. The question is "can the beat +# see this", not "how loudly", so a handful of examples is the whole evidence a +# reader needs and an unbounded list would bury the verdict. +DEFAULT_MATCH_LIMIT = 5 + + +@dataclass(frozen=True) +class CompiledTerm: + """One probe term as written, with its compiled pattern.""" + + raw: str + pattern: re.Pattern[str] + + def found_in(self, text: str) -> bool: + return bool(self.pattern.search(text)) + + +def _compile_term(raw: str) -> CompiledTerm: + """ + Compile one term under the same boundary and case rules as coverage entities. + + A term written entirely in capitals is an acronym and matches + case-sensitively -- ``BOD`` is a directive, ``bod`` is not. Everything else + matches case-insensitively, because ordinary names are case-varied in prose + and long enough not to collide. + """ + stem = raw.endswith(STEM_MARKER) + body = raw[: -len(STEM_MARKER)] if stem else raw + source = term_pattern(body, right_boundary=not stem) + flags = 0 if is_shouted(body) else re.IGNORECASE + return CompiledTerm(raw=raw, pattern=re.compile(source, flags)) + + +@dataclass(frozen=True) +class CompiledProbe: + """ + One probe with every term compiled, ready to be run against many articles. + + ``required`` must all be found in the same article; each group in ``groups`` + contributes one alternative that must be found. An article satisfies the + probe only when both hold. + """ + + probe: CoverageProbe + required: tuple[CompiledTerm, ...] + groups: tuple[tuple[CompiledTerm, ...], ...] + + def matches(self, text: str) -> bool: + """Whether one article's text satisfies every requirement.""" + absent, unsatisfied = self.missing(text) + return not absent and not unsatisfied + + def missing(self, text: str) -> tuple[tuple[str, ...], tuple[tuple[str, ...], ...]]: + """ + What this text failed to supply: absent required terms, then unsatisfied + groups. + + Returned rather than swallowed so an unmatched probe can say *which* + requirement went unmet. An operator reading "FedRAMP absent" fixes a + source; one reading "matched FedRAMP but nothing in (director, + administrator)" fixes the probe. + """ + absent = tuple(term.raw for term in self.required if not term.found_in(text)) + unsatisfied = tuple( + tuple(term.raw for term in group) + for group in self.groups + if not any(term.found_in(text) for term in group) + ) + return absent, unsatisfied + + +def compile_probe(probe: CoverageProbe) -> CompiledProbe: + """Compile one probe's terms once, for reuse across every candidate article.""" + return CompiledProbe( + probe=probe, + required=tuple(_compile_term(term) for term in probe.terms), + groups=tuple(tuple(_compile_term(term) for term in group) for group in probe.any_of), + ) + + +@dataclass(frozen=True) +class ProbeMatch: + """One article that satisfied a probe, and the source that carried it.""" + + article_id: int + title: str + feed_name: str + published_date: datetime | None + in_beat_sources: bool + + @property + def published_day(self) -> str: + return self.published_date.date().isoformat() if self.published_date else "undated" + + +@dataclass(frozen=True) +class ProbeResult: + """ + What one probe found. + + ``elsewhere`` is only populated for a probe the beat's own sources missed: + it names articles from feeds *outside* this beat that would have satisfied + it. That distinction is the actionable part. "Nobody in the corpus carried + it" and "three feeds you do not subscribe to carried it" are the same + ``UNMATCHED`` verdict and completely different repairs. + """ + + probe: CoverageProbe + status: str + window: tuple[date, date] + articles_in_window: int + matches: tuple[ProbeMatch, ...] = () + match_count: int = 0 + elsewhere: tuple[ProbeMatch, ...] = () + elsewhere_count: int = 0 + reason: str = "" + missing_terms: tuple[str, ...] = () + unsatisfied_groups: tuple[tuple[str, ...], ...] = () + + @property + def matched(self) -> bool: + return self.status == STATUS_MATCHED + + @property + def conclusive(self) -> bool: + return self.status != STATUS_INCONCLUSIVE + + +@dataclass(frozen=True) +class CoverageReport: + """ + Every probe's result for one beat, plus the corpus facts that framed them. + + ``total`` is the number of probes the beat *declared*, and the passed, + unmatched and inconclusive counts always sum to it. The invariant is the + point: an inconclusive probe is subtracted from neither side, so a probe set + rotting out of retention shows up as a shrinking denominator of *answered* + questions rather than as a clean sweep of the ones that remain. + """ + + beat_name: str + results: tuple[ProbeResult, ...] + feed_names: tuple[str, ...] + earliest: datetime | None + latest: datetime | None + + @property + def total(self) -> int: + return len(self.results) + + @property + def matched(self) -> tuple[ProbeResult, ...]: + return tuple(r for r in self.results if r.status == STATUS_MATCHED) + + @property + def unmatched(self) -> tuple[ProbeResult, ...]: + return tuple(r for r in self.results if r.status == STATUS_UNMATCHED) + + @property + def inconclusive(self) -> tuple[ProbeResult, ...]: + return tuple(r for r in self.results if r.status == STATUS_INCONCLUSIVE) + + @property + def conclusive(self) -> tuple[ProbeResult, ...]: + return tuple(r for r in self.results if r.conclusive) + + +def article_text(article: Article) -> str: + """ + The text of one stored article that probe matching reads. + + Title, summary and normalized body. ``author`` is excluded, as it is in + :func:`src.context.entity_matcher.item_text`: a byline is a person, and a + probe is about whether a source carried an event, never about who wrote it. + """ + parts = [ + article.title or "", + article.description or "", + article.normalized_content or article.content or "", + ] + return "\n".join(part for part in parts if part) + + +def _window_bounds(probe: CoverageProbe) -> tuple[datetime, datetime]: + """The probe's date window as the half-open datetime range SQL compares.""" + start, end = probe.window + return datetime.combine(start, time.min), datetime.combine(end, time.max) + + +def _to_match(article: Article, feed_name: str | None, in_beat_sources: bool) -> ProbeMatch: + return ProbeMatch( + article_id=article.id, + title=article.title or "(untitled)", + feed_name=feed_name or "(unattributed source)", + published_date=article.published_date, + in_beat_sources=in_beat_sources, + ) + + +def _rows_in_window( + session: Session, probe: CoverageProbe, feed_ids: Sequence[int] | None +) -> list[tuple[Article, str | None]]: + """ + Every dated article in the probe's window, with its source name. + + Undated rows are excluded rather than assumed recent: an article with no + ``published_date`` cannot be placed in or out of any window, and counting it + as in-window would let an unplaceable row turn an inconclusive probe into a + failure. + + This is the only database access in the module and it is a SELECT. Nothing + in the coverage path writes -- the corpus is shared, and a read-only + question must not leave a trace in the thing it is asking about. + """ + start, end = _window_bounds(probe) + query = ( + session.query(Article, RSSFeed.name) + .outerjoin(RSSFeed, Article.feed_id == RSSFeed.id) + .filter(Article.published_date.isnot(None)) + .filter(Article.published_date >= start) + .filter(Article.published_date <= end) + ) + if feed_ids is not None: + query = query.filter(Article.feed_id.in_(feed_ids)) + return query.order_by(Article.published_date.asc()).all() + + +@dataclass(frozen=True) +class _Evaluation: + """One probe's outcome over one set of candidate rows.""" + + matches: tuple[ProbeMatch, ...] + match_count: int + missing_terms: tuple[str, ...] + unsatisfied_groups: tuple[tuple[str, ...], ...] + + +def _evaluate( + compiled: CompiledProbe, + rows: Iterable[tuple[Article, str | None]], + in_beat_sources: bool, + limit: int, +) -> _Evaluation: + """ + Run one compiled probe over candidate rows. + + ``match_count`` is every match; ``matches`` is the earliest ``limit`` of + them, kept separate so the display cap can never make a widely-reported + event look like a single lucky hit. + + Also returns the *closest* near miss -- the requirements unmet by whichever + article satisfied the most of them. With no matches that is the diagnostic + an operator acts on; with matches it is discarded by the caller. + """ + matches: list[ProbeMatch] = [] + match_count = 0 + best_absent: tuple[str, ...] = tuple(term.raw for term in compiled.required) + best_groups: tuple[tuple[str, ...], ...] = tuple( + tuple(term.raw for term in group) for group in compiled.groups + ) + best_score = len(best_absent) + len(best_groups) + + for article, feed_name in rows: + absent, unsatisfied = compiled.missing(article_text(article)) + if not absent and not unsatisfied: + match_count += 1 + if len(matches) < limit: + matches.append(_to_match(article, feed_name, in_beat_sources)) + continue + score = len(absent) + len(unsatisfied) + if score < best_score: + best_score, best_absent, best_groups = score, absent, unsatisfied + + return _Evaluation(tuple(matches), match_count, best_absent, best_groups) + + +def run_coverage_probes( + session: Session, + beat: BeatConfig, + feed_urls: Sequence[str] | None = None, + match_limit: int = DEFAULT_MATCH_LIMIT, +) -> CoverageReport: + """ + Run every probe a beat declares against the corpus, read-only. + + The verdict is scoped to the beat's own sources: a probe passes when a feed + *this beat subscribes to* carried the event. An unmatched probe is then + re-run across the whole corpus, and any hit is reported as ``elsewhere`` -- + evidence that the event was reachable and this beat's source list is what + missed it. + + A beat with no probes returns an empty report. That is not a pass, and the + caller is expected to say so: an undeclared probe set is the state the beat + was already in when it missed the FedRAMP reinstatement. + """ + feeds = beat.resolve_feeds() + urls = list(feed_urls) if feed_urls is not None else [feed.url for feed in feeds] + feed_rows = session.query(RSSFeed).filter(RSSFeed.url.in_(urls)).all() if urls else [] + feed_ids = [row.id for row in feed_rows] + feed_names = tuple(sorted(row.name for row in feed_rows)) + + earliest, latest = _corpus_bounds(session, feed_ids) + + results: list[ProbeResult] = [] + for probe in beat.coverage_probes: + compiled = compile_probe(probe) + rows = _rows_in_window(session, probe, feed_ids) if feed_ids else [] + found = _evaluate(compiled, rows, True, match_limit) + + if found.match_count: + results.append( + ProbeResult( + probe=probe, + status=STATUS_MATCHED, + window=probe.window, + articles_in_window=len(rows), + matches=found.matches, + match_count=found.match_count, + ) + ) + continue + + if not rows: + results.append( + ProbeResult( + probe=probe, + status=STATUS_INCONCLUSIVE, + window=probe.window, + articles_in_window=0, + reason=_inconclusive_reason(probe, earliest, latest, bool(feed_ids)), + ) + ) + continue + + searched_ids = {article.id for article, _ in rows} + wider = _evaluate(compiled, _rows_in_window(session, probe, None), False, match_limit) + elsewhere = tuple(m for m in wider.matches if m.article_id not in searched_ids) + results.append( + ProbeResult( + probe=probe, + status=STATUS_UNMATCHED, + window=probe.window, + articles_in_window=len(rows), + elsewhere=elsewhere, + elsewhere_count=wider.match_count, + missing_terms=found.missing_terms, + unsatisfied_groups=found.unsatisfied_groups, + ) + ) + + return CoverageReport( + beat_name=beat.name, + results=tuple(results), + feed_names=feed_names, + earliest=earliest, + latest=latest, + ) + + +def _corpus_bounds( + session: Session, feed_ids: Sequence[int] +) -> tuple[datetime | None, datetime | None]: + """Oldest and newest dated article carried by the beat's own sources.""" + if not feed_ids: + return None, None + query = session.query(Article.published_date).filter( + Article.feed_id.in_(feed_ids), Article.published_date.isnot(None) + ) + earliest = query.order_by(Article.published_date.asc()).first() + latest = query.order_by(Article.published_date.desc()).first() + return (earliest[0] if earliest else None), (latest[0] if latest else None) + + +def _inconclusive_reason( + probe: CoverageProbe, + earliest: datetime | None, + latest: datetime | None, + has_feeds: bool, +) -> str: + """ + Why a probe could not be answered, in the terms an operator can act on. + + The three cases are different problems. Predating the corpus means the probe + has aged out and should be replaced. Postdating it means the pipeline has not + run far enough forward yet. A gap in the middle means ingestion was down, and + that is worth knowing on its own. + """ + if not has_feeds: + return "this beat's sources have no rows in the corpus, so nothing was searched" + if earliest is None: + return "this beat's sources have stored no dated articles, so nothing was searched" + start, end = probe.window + if end < earliest.date(): + return ( + f"window ends {end.isoformat()}, before the beat's oldest stored article " + f"({earliest.date().isoformat()}) -- the event predates the corpus" + ) + if latest is not None and start > latest.date(): + return ( + f"window starts {start.isoformat()}, after the beat's newest stored article " + f"({latest.date().isoformat()}) -- the corpus has not reached the event yet" + ) + return ( + "the beat's sources stored no dated articles anywhere in this window -- " + "a gap in the corpus, not a gap in the beat" + ) diff --git a/src/context/entity_matcher.py b/src/context/entity_matcher.py index 864dcaa..c92b819 100644 --- a/src/context/entity_matcher.py +++ b/src/context/entity_matcher.py @@ -35,23 +35,31 @@ from ..config.beats import CoverageEntity __all__ = [ + "LEFT_BOUNDARY", + "RIGHT_BOUNDARY", "CompiledEntity", "compile_entities", "count_item_mentions", + "is_shouted", "item_text", + "term_pattern", ] # What counts as "inside a word". Deliberately alphanumerics only: a hyphen or # a slash is a boundary, so "CISA-issued" and "FedRAMP/StateRAMP" are mentions, # while "precisa" and "bombing" are not. _WORD_CHAR = "[0-9A-Za-z]" -_LEFT_BOUNDARY = f"(? bool: + +def is_shouted(term: str) -> bool: """ True when a term is written in capitals only -- i.e. it is an acronym. @@ -63,15 +71,21 @@ def _is_shouted(term: str) -> bool: return bool(letters) and all(char.isupper() for char in letters) -def _term_pattern(term: str) -> str: +def term_pattern(term: str, right_boundary: bool = True) -> str: """ Regex source for one surface form. Runs of whitespace in the term match any run of whitespace in the text, so a name broken across a line wrap in the source still matches. + + ``right_boundary=False`` anchors only the left-hand side, which turns the + term into a stem: it must still begin a word, but the word may continue. + Coverage probes use it for ``reinstat*``; coverage entities never do, + because an entity's surface forms are whole names. """ body = _WHITESPACE.join(re.escape(token) for token in term.split()) - return f"{_LEFT_BOUNDARY}(?:{body}){_RIGHT_BOUNDARY}" + tail = RIGHT_BOUNDARY if right_boundary else "" + return f"{LEFT_BOUNDARY}(?:{body}){tail}" @dataclass(frozen=True) @@ -111,7 +125,7 @@ def compile_entities(entities: Iterable[CoverageEntity]) -> list[CompiledEntity] for term in entity.terms: if not term.strip(): continue - (shouted if _is_shouted(term) else relaxed).append(_term_pattern(term)) + (shouted if is_shouted(term) else relaxed).append(term_pattern(term)) patterns: list[re.Pattern[str]] = [] if shouted: diff --git a/tests/cli/test_beat_coverage.py b/tests/cli/test_beat_coverage.py new file mode 100644 index 0000000..871a77a --- /dev/null +++ b/tests/cli/test_beat_coverage.py @@ -0,0 +1,304 @@ +""" +Tests for ``insightweaver beat coverage NAME``. + +What is pinned here is the gate: the exit code, and the fact that the counts +printed under it always add back up to the number of probes the beat declared. +The command is only useful if a shell can trust it, and the two ways to break +that trust are exiting 0 on a miss and exiting 0 on a run that measured nothing. + +Every beat file and every article here is written by the test. No test names a +real event or reads the shipped beat's probes -- those are statements about live +corpus state, and pinning them into CI would make this suite fail for reasons +unrelated to the code under test. + +Also pinned: the command runs with no ANTHROPIC_API_KEY and never writes. +""" + +import json +from datetime import datetime + +import pytest +from sqlalchemy import create_engine +from sqlalchemy.orm import sessionmaker + +from src.cli.beat import ( + EXIT_NOTHING_MEASURED, + EXIT_OK, + EXIT_UNMATCHED, + beat_command, +) +from src.config.beats import load_beat +from src.config.feed_matcher import Feed +from src.database.models import Article, Base, RSSFeed + +FEED_URL = "https://example.test/in-beat" +FEED_NAME = "In-Beat Wire" + +MATCHING_PROBE = { + "date": "2026-08-24", + "what": "the thing that happened", + "terms": ["FedRAMP"], + "any_of": [["director", "administrator"], ["reinstat*", "restored"]], +} +MISSING_PROBE = { + "date": "2026-08-24", + "what": "the thing nobody carried", + "terms": ["CMMC"], + "any_of": [["phase 2", "phase two"]], +} +OLD_PROBE = { + "date": "2019-01-15", + "what": "the thing that predates the corpus", + "terms": ["CISA"], + "any_of": [["binding operational directive"]], +} + + +@pytest.fixture +def beats_dir(tmp_path, monkeypatch): + """A beats directory the test owns, wired into the command.""" + directory = tmp_path / "beats" + directory.mkdir() + monkeypatch.setattr( + "src.cli.beat.load_beat", lambda name, beats_dir=directory: load_beat(name, beats_dir) + ) + return directory + + +@pytest.fixture +def write_beat(beats_dir): + def _write(probes, name="probe-beat"): + (beats_dir / f"{name}.json").write_text( + json.dumps( + { + "name": name, + "description": "A beat under test.", + "sources": [{"adapter": "rss", "feed_tags": ["regulatory"]}], + "coverage_probes": probes, + } + ) + ) + return name + + return _write + + +@pytest.fixture +def fake_feeds(monkeypatch): + """One configured feed, so the beat resolves without reading config/feeds.""" + + class FakeMatcher: + all_feeds = [ + Feed( + name=FEED_NAME, + url=FEED_URL, + scope=[], + geo_tags=[], + domain_tags=["regulatory"], + specialty_tags=[], + relevance_score=0.9, + source_file="fake.json", + ) + ] + + monkeypatch.setattr("src.config.beats.FeedMatcher", FakeMatcher) + + +@pytest.fixture +def corpus(tmp_path, monkeypatch): + """ + A throwaway database wired into the command, plus a helper to fill it. + + ``SessionLocal`` is replaced rather than the global engine so that no test + here depends on whatever DATABASE_URL happens to name. + """ + engine = create_engine(f"sqlite:///{tmp_path / 'coverage.db'}") + Base.metadata.create_all(bind=engine) + maker = sessionmaker(bind=engine) + monkeypatch.setattr("src.cli.beat.SessionLocal", maker) + + session = maker() + feed = RSSFeed(url=FEED_URL, name=FEED_NAME, category="test") + session.add(feed) + session.commit() + + def add(title, when=datetime(2026, 8, 25)): + session.add( + Article( + feed_id=feed.id, + guid=f"guid-{title}", + title=title, + description="", + normalized_content="", + published_date=when, + ) + ) + session.commit() + + add.session = session + add.engine = engine + yield add + session.close() + + +@pytest.fixture(autouse=True) +def no_api_key(monkeypatch): + """The command must work with no key at all; this is not optional.""" + monkeypatch.delenv("ANTHROPIC_API_KEY", raising=False) + + +def invoke(cli_runner, name): + return cli_runner.invoke(beat_command, ["coverage", name]) + + +class TestTheGate: + def test_all_probes_matched_exits_zero(self, cli_runner, write_beat, fake_feeds, corpus): + corpus("FedRAMP director reinstated after review") + result = invoke(cli_runner, write_beat([MATCHING_PROBE])) + + assert result.exit_code == EXIT_OK + assert "[MATCHED]" in result.output + assert "PASS" in result.output + + def test_an_unmatched_probe_exits_non_zero(self, cli_runner, write_beat, fake_feeds, corpus): + corpus("An unrelated procurement notice") + result = invoke(cli_runner, write_beat([MISSING_PROBE])) + + assert result.exit_code == EXIT_UNMATCHED + assert result.exit_code != 0 + assert "[UNMATCHED]" in result.output + assert "FAIL" in result.output + + def test_one_miss_among_matches_still_fails(self, cli_runner, write_beat, fake_feeds, corpus): + corpus("FedRAMP director reinstated after review") + result = invoke(cli_runner, write_beat([MATCHING_PROBE, MISSING_PROBE])) + + assert result.exit_code == EXIT_UNMATCHED + assert "[MATCHED]" in result.output + assert "[UNMATCHED]" in result.output + + def test_a_beat_with_no_probes_is_not_a_pass(self, cli_runner, write_beat, fake_feeds, corpus): + """ + An undeclared probe set is the state the beat was in when it missed the + event this feature exists because of. It must not exit 0. + """ + result = invoke(cli_runner, write_beat([])) + + assert result.exit_code == EXIT_NOTHING_MEASURED + assert "declares no coverage_probes" in result.output + + def test_every_probe_inconclusive_is_not_a_pass_either( + self, cli_runner, write_beat, fake_feeds, corpus + ): + corpus("Recent unrelated news") + result = invoke(cli_runner, write_beat([OLD_PROBE])) + + assert result.exit_code == EXIT_NOTHING_MEASURED + assert "[INCONCLUSIVE]" in result.output + assert "NOT MEASURED" in result.output + + def test_the_three_exit_codes_are_distinct(self): + assert len({EXIT_OK, EXIT_UNMATCHED, EXIT_NOTHING_MEASURED}) == 3 + + +class TestOutput: + def test_a_match_names_the_feed_that_carried_it( + self, cli_runner, write_beat, fake_feeds, corpus + ): + corpus("FedRAMP director reinstated after review") + result = invoke(cli_runner, write_beat([MATCHING_PROBE])) + assert FEED_NAME in result.output + + def test_an_unmatched_probe_says_what_the_closest_article_lacked( + self, cli_runner, write_beat, fake_feeds, corpus + ): + corpus("CMMC rulemaking continues") + result = invoke(cli_runner, write_beat([MISSING_PROBE])) + assert "phase 2" in result.output + + def test_an_inconclusive_probe_says_why(self, cli_runner, write_beat, fake_feeds, corpus): + corpus("Recent unrelated news") + result = invoke(cli_runner, write_beat([OLD_PROBE])) + assert "predates the corpus" in result.output + + def test_the_denominator_is_every_probe_declared( + self, cli_runner, write_beat, fake_feeds, corpus + ): + """ + The inconclusive probe stays in the count. A suite that decayed to only + the probes still matching would print '1 matched ... of 1 declared'. + """ + corpus("FedRAMP director reinstated after review") + result = invoke(cli_runner, write_beat([MATCHING_PROBE, OLD_PROBE])) + + assert "1 matched + 0 unmatched + 1 inconclusive of 2 probe(s) declared" in result.output + assert result.exit_code == EXIT_OK + assert "counted above, not dropped" in result.output + + def test_show_problems_hides_the_passes_only(self, cli_runner, write_beat, fake_feeds, corpus): + corpus("FedRAMP director reinstated after review") + result = cli_runner.invoke( + beat_command, + ["coverage", write_beat([MATCHING_PROBE, OLD_PROBE]), "--show", "problems"], + ) + assert "the thing that happened" not in result.output + assert "the thing that predates the corpus" in result.output + assert "of 2 probe(s) declared" in result.output + + +class TestFailures: + def test_an_unknown_beat_is_an_error_not_a_pass(self, cli_runner, beats_dir, corpus): + result = invoke(cli_runner, "no-such-beat") + assert result.exit_code != 0 + assert "No beat named" in result.output + + def test_an_invalid_probe_block_is_reported_by_name(self, cli_runner, beats_dir, corpus): + (beats_dir / "broken.json").write_text( + json.dumps( + { + "name": "broken", + "sources": [{"adapter": "rss", "feed_tags": ["regulatory"]}], + "coverage_probes": [{"date": "2026-08-24", "what": "x", "terms": ["FedRAMP"]}], + } + ) + ) + result = invoke(cli_runner, "broken") + assert result.exit_code != 0 + assert "single term" in result.output + + +class TestItNeverWrites: + def test_the_corpus_is_unchanged_by_a_run(self, cli_runner, write_beat, fake_feeds, corpus): + corpus("FedRAMP director reinstated after review") + corpus("An unrelated procurement notice") + before = corpus.session.query(Article).count() + + invoke(cli_runner, write_beat([MATCHING_PROBE, MISSING_PROBE, OLD_PROBE])) + + corpus.session.expire_all() + assert corpus.session.query(Article).count() == before + + def test_the_command_never_commits( + self, cli_runner, write_beat, fake_feeds, corpus, monkeypatch + ): + """ + The corpus is shared. Committing nothing is still an intent to write, so + the read path must not reach for commit at all. + """ + maker = sessionmaker(bind=corpus.engine) + commits = [] + + class Watched(maker.class_): + def commit(self, *args, **kwargs): + commits.append(True) + raise AssertionError("beat coverage must not commit") + + monkeypatch.setattr( + "src.cli.beat.SessionLocal", sessionmaker(bind=corpus.engine, class_=Watched) + ) + corpus("FedRAMP director reinstated after review") + + result = invoke(cli_runner, write_beat([MATCHING_PROBE])) + + assert commits == [] + assert result.exit_code == EXIT_OK diff --git a/tests/config/test_beat_probes.py b/tests/config/test_beat_probes.py new file mode 100644 index 0000000..7743724 --- /dev/null +++ b/tests/config/test_beat_probes.py @@ -0,0 +1,194 @@ +""" +Tests for the ``coverage_probes`` block of a beat file. + +A probe is a claim that a match means the beat saw a specific event. The loader +is therefore the first place that claim can be weakened, so what is pinned here +is mostly refusal: a probe is rejected rather than repaired, and in particular a +probe resting on one bare term is rejected outright. A probe that passes on +generic terms manufactures exactly the confidence backlog task 010 exists to +remove. + +Every fixture here writes its own beat file. Nothing reads the shipped beat's +probe list, because that list is about live corpus state and would make this +suite fail for reasons unrelated to the loader. +""" + +import json +from datetime import date + +import pytest + +from src.config.beats import ( + DEFAULT_PROBE_WINDOW_DAYS, + MIN_PROBE_EVIDENCE, + BeatValidationError, + load_beat, +) + + +@pytest.fixture +def beats_dir(tmp_path): + directory = tmp_path / "beats" + directory.mkdir() + return directory + + +def write(beats_dir, probes, name="probe-beat"): + payload = { + "name": name, + "description": "A beat with probes.", + "sources": [{"adapter": "rss", "feed_tags": ["regulatory"]}], + "coverage_probes": probes, + } + (beats_dir / f"{name}.json").write_text(json.dumps(payload)) + return name + + +def load(beats_dir, probes): + return load_beat(write(beats_dir, probes), beats_dir) + + +VALID = { + "date": "2026-08-24", + "what": "FedRAMP director reinstated", + "terms": ["FedRAMP"], + "any_of": [["director", "administrator"], ["reinstat*", "return*", "restored"]], +} + + +class TestProbesLoad: + def test_a_beat_may_declare_no_probes_at_all(self, beats_dir): + assert load_beat(write(beats_dir, []), beats_dir).coverage_probes == () + + def test_the_block_is_optional(self, beats_dir): + name = "no-probe-key" + (beats_dir / f"{name}.json").write_text( + json.dumps({"name": name, "sources": [{"adapter": "rss", "feed_tags": ["regulatory"]}]}) + ) + assert load_beat(name, beats_dir).coverage_probes == () + + def test_a_probe_carries_a_date_a_description_and_its_terms(self, beats_dir): + (probe,) = load(beats_dir, [VALID]).coverage_probes + assert probe.date == date(2026, 8, 24) + assert probe.what == "FedRAMP director reinstated" + assert probe.terms == ("FedRAMP",) + assert probe.any_of == ( + ("director", "administrator"), + ("reinstat*", "return*", "restored"), + ) + + def test_the_window_defaults_and_is_symmetric_around_the_event(self, beats_dir): + (probe,) = load(beats_dir, [VALID]).coverage_probes + assert probe.window_days == DEFAULT_PROBE_WINDOW_DAYS + assert probe.window == (date(2026, 8, 10), date(2026, 9, 7)) + + def test_the_window_can_be_widened_per_probe(self, beats_dir): + (probe,) = load(beats_dir, [{**VALID, "window_days": 1}]).coverage_probes + assert probe.window == (date(2026, 8, 23), date(2026, 8, 25)) + + def test_terms_are_deduplicated_but_keep_their_order(self, beats_dir): + (probe,) = load( + beats_dir, [{**VALID, "terms": ["FedRAMP", "PMO", "FedRAMP"]}] + ).coverage_probes + assert probe.terms == ("FedRAMP", "PMO") + + def test_describe_reads_back_the_requirement_as_written(self, beats_dir): + (probe,) = load(beats_dir, [VALID]).coverage_probes + assert probe.describe() == ( + "FedRAMP AND (director OR administrator) AND (reinstat* OR return* OR restored)" + ) + + +class TestASingleGenericTermIsRefused: + """ + The load-bearing refusal. + + `FedRAMP` on its own is matched by an AWS region-launch post. A probe that + would pass on that is worse than no probe, because a green tick from it is + read as coverage. + """ + + def test_one_term_and_no_groups_is_rejected(self, beats_dir): + with pytest.raises(BeatValidationError) as exc: + load(beats_dir, [{"date": "2026-08-24", "what": "x", "terms": ["FedRAMP"]}]) + assert "single term" in str(exc.value) + assert str(MIN_PROBE_EVIDENCE) in str(exc.value) + + def test_two_required_terms_is_enough(self, beats_dir): + probes = load( + beats_dir, [{"date": "2026-08-24", "what": "x", "terms": ["FedRAMP", "PMO"]}] + ).coverage_probes + assert probes[0].evidence_count == 2 + + def test_one_term_plus_one_group_is_enough(self, beats_dir): + probes = load( + beats_dir, + [ + { + "date": "2026-08-24", + "what": "x", + "terms": ["FedRAMP"], + "any_of": [["director"]], + } + ], + ).coverage_probes + assert probes[0].evidence_count == 2 + + def test_a_repeated_term_does_not_count_twice(self, beats_dir): + """Deduplication happens before the floor is checked, not after.""" + with pytest.raises(BeatValidationError, match="single term"): + load( + beats_dir, + [{"date": "2026-08-24", "what": "x", "terms": ["FedRAMP", "FedRAMP"]}], + ) + + +class TestRejections: + def test_unknown_probe_key(self, beats_dir): + with pytest.raises(BeatValidationError, match="unknown key"): + load(beats_dir, [{**VALID, "author": "someone"}]) + + def test_missing_date(self, beats_dir): + with pytest.raises(BeatValidationError, match="missing required key"): + load(beats_dir, [{"what": "x", "terms": ["a", "b"]}]) + + def test_unparseable_date_is_refused_not_defaulted(self, beats_dir): + with pytest.raises(BeatValidationError, match="not a 'YYYY-MM-DD' date"): + load(beats_dir, [{**VALID, "date": "last August"}]) + + def test_empty_description(self, beats_dir): + with pytest.raises(BeatValidationError, match="non-empty description"): + load(beats_dir, [{**VALID, "what": " "}]) + + def test_empty_terms(self, beats_dir): + with pytest.raises(BeatValidationError, match="terms must not be empty"): + load(beats_dir, [{**VALID, "terms": []}]) + + def test_an_empty_any_of_group_is_refused(self, beats_dir): + """An empty group is satisfied by nothing, so it would silently kill the probe.""" + with pytest.raises(BeatValidationError, match="must not be empty"): + load(beats_dir, [{**VALID, "any_of": [["director"], []]}]) + + def test_a_bare_stem_marker_is_refused(self, beats_dir): + with pytest.raises(BeatValidationError, match="stem marker with no stem"): + load(beats_dir, [{**VALID, "terms": ["FedRAMP", "*"]}]) + + def test_non_string_term(self, beats_dir): + with pytest.raises(BeatValidationError, match="only non-empty strings"): + load(beats_dir, [{**VALID, "terms": ["FedRAMP", 7]}]) + + def test_window_days_must_be_a_positive_integer(self, beats_dir): + with pytest.raises(BeatValidationError, match="positive integer"): + load(beats_dir, [{**VALID, "window_days": 0}]) + + def test_window_days_rejects_a_bool(self, beats_dir): + with pytest.raises(BeatValidationError, match="positive integer"): + load(beats_dir, [{**VALID, "window_days": True}]) + + def test_probe_list_must_be_a_list(self, beats_dir): + with pytest.raises(BeatValidationError, match="must be a list"): + load(beats_dir, {"date": "2026-08-24"}) + + def test_probe_must_be_an_object(self, beats_dir): + with pytest.raises(BeatValidationError, match="must be an object"): + load(beats_dir, ["FedRAMP director reinstated"]) diff --git a/tests/context/test_coverage_probe.py b/tests/context/test_coverage_probe.py new file mode 100644 index 0000000..3327b99 --- /dev/null +++ b/tests/context/test_coverage_probe.py @@ -0,0 +1,403 @@ +""" +Tests for running a beat's coverage probes against the corpus. + +Every corpus here is synthetic and built by the test that reads it. None of it +touches the live database, and no test names an event that has to still be in +retention: this is a test of the matcher and the verdict logic, not of what the +operator's corpus happens to hold today. Pinning live events into CI would make +the suite fail for reasons that have nothing to do with the code under test. + +The collision surface is the point of interest, and it is measured rather than +asserted from intuition. On this repository's corpus on 2026-08-27, of 54,044 +stored articles 657 titles contain the substring ``nist`` and exactly 4 are +about NIST. :class:`TestSubstringMatchingWouldLie` reproduces that ratio in +miniature and shows both matchers side by side. +""" + +from datetime import date, datetime + +import pytest +from sqlalchemy.orm import sessionmaker + +from src.config.beats import BeatConfig, BeatSource, CoverageProbe +from src.config.feed_matcher import Feed +from src.context.coverage_probe import ( + STATUS_INCONCLUSIVE, + STATUS_MATCHED, + STATUS_UNMATCHED, + compile_probe, + run_coverage_probes, +) +from src.database.models import Article, RSSFeed + +# Titles that contain "nist" as a substring and are about nothing of the kind. +# Drawn from the words the real collision was made of: administration, minister, +# Afghanistan, communist, sinister, columnist, agonist. +NIST_COLLISIONS = [ + "The administration reshuffles its priorities", + "Prime minister addresses the assembly", + "Afghanistan aid package clears committee", + "A communist party congress opens", + "A sinister turn in the negotiations", + "Our columnist on the week in politics", + "Beta agonist trial results published", + "Administrative law judges face a backlog", + "Ministry of Defence confirms the order", +] +NIST_REAL = [ + "NIST is rethinking its role in analyzing software vulnerabilities", + "NIST and MITRE partner to test AI defense technology", +] + + +def probe(**overrides): + fields = { + "date": date(2026, 8, 24), + "what": "an event", + "terms": ("NIST",), + "any_of": (("vulnerabilit*", "partner*"),), + "window_days": 14, + } + fields.update(overrides) + return CoverageProbe(**fields) + + +def text_of(title): + return title + + +class TestTermMatching: + """Pure matching, no database.""" + + def test_a_required_term_must_be_present(self): + compiled = compile_probe(probe(terms=("FedRAMP", "PMO"), any_of=())) + assert compiled.matches("The FedRAMP PMO published guidance") + assert not compiled.matches("The FedRAMP program published guidance") + + def test_each_any_of_group_needs_one_member(self): + compiled = compile_probe( + probe(terms=("FedRAMP",), any_of=(("director", "administrator"), ("reinstat*",))) + ) + assert compiled.matches("FedRAMP director reinstated after review") + assert compiled.matches("FedRAMP administrator reinstatement confirmed") + assert not compiled.matches("FedRAMP director named after review") + + def test_missing_reports_which_requirement_went_unmet(self): + compiled = compile_probe(probe(terms=("FedRAMP",), any_of=(("director", "administrator"),))) + absent, unsatisfied = compiled.missing("CISA director named") + assert absent == ("FedRAMP",) + assert unsatisfied == () + + absent, unsatisfied = compiled.missing("FedRAMP authorizations paused") + assert absent == () + assert unsatisfied == (("director", "administrator"),) + + +class TestSubstringMatchingWouldLie: + """ + The word-boundary requirement, shown biting. + + Both matchers are run over the same nine collisions and two real articles. + The substring matcher reports 11 of 11; the probe matcher reports 2 of 11. + """ + + def test_substring_matching_reports_every_collision(self): + naive = [t for t in NIST_COLLISIONS + NIST_REAL if "nist" in t.lower()] + assert len(naive) == 11 + + def test_probe_matching_reports_only_the_real_ones(self): + compiled = compile_probe(probe(terms=("NIST",), any_of=(("rethinking", "partner*"),))) + matched = [t for t in NIST_COLLISIONS + NIST_REAL if compiled.matches(text_of(t))] + assert matched == NIST_REAL + + @pytest.mark.parametrize("title", NIST_COLLISIONS) + def test_no_collision_matches_even_the_bare_term(self, title): + """The boundary, not the two-level shape, is what rejects these.""" + compiled = compile_probe(probe(terms=("NIST", "the"), any_of=())) + assert not compiled.matches(title) + + def test_an_acronym_only_matches_when_it_is_shouted(self): + """Inherited from entity matching: BOD is a directive, bod is not.""" + compiled = compile_probe(probe(terms=("BOD", "issued"), any_of=())) + assert compiled.matches("CISA issued BOD 22-01") + assert not compiled.matches("A bod issued a statement") + + +class TestStems: + def test_a_stem_must_be_marked(self): + assert not compile_probe(probe(terms=("reinstat", "x"), any_of=())).matches( + "x reinstated today" + ) + assert compile_probe(probe(terms=("reinstat*", "x"), any_of=())).matches( + "x reinstated today" + ) + + def test_a_stem_still_has_to_start_a_word(self): + """The left boundary is what does the collision filtering; a stem keeps it.""" + compiled = compile_probe(probe(terms=("ramp*", "the"), any_of=())) + assert not compiled.matches("the FedRAMP authorization") + assert compiled.matches("the ramp-up of authorizations") + + +# --------------------------------------------------------------------------- +# Corpus-backed verdicts +# --------------------------------------------------------------------------- + +FEED_URL = "https://example.test/beat-feed" +FEED_NAME = "In-Beat Wire" +OTHER_URL = "https://example.test/other-feed" +OTHER_NAME = "Out-of-Beat Wire" + + +@pytest.fixture +def session(test_engine): + maker = sessionmaker(bind=test_engine) + db = maker() + yield db + db.close() + + +@pytest.fixture +def corpus(session): + """Two sources, empty. Tests add the articles they need.""" + in_beat = RSSFeed(url=FEED_URL, name=FEED_NAME, category="test") + outside = RSSFeed(url=OTHER_URL, name=OTHER_NAME, category="test") + session.add_all([in_beat, outside]) + session.commit() + return in_beat, outside + + +def add_article(session, feed, title, when, body=""): + session.add( + Article( + feed_id=feed.id, + guid=f"{feed.id}-{title}", + title=title, + description="", + normalized_content=body, + published_date=when, + ) + ) + session.commit() + + +def beat_with(probes, feed_urls=(FEED_URL,)): + return BeatConfig( + name="test-beat", + description="", + sources=(BeatSource(adapter="rss", feed_tags=("regulatory",)),), + coverage={}, + standing_questions=(), + channels=("terminal",), + config_path="", + coverage_probes=tuple(probes), + ), list(feed_urls) + + +def run(session, probes, feed_urls=(FEED_URL,)): + beat, urls = beat_with(probes, feed_urls) + return run_coverage_probes(session, beat, feed_urls=urls) + + +class TestVerdicts: + def test_a_match_names_the_feed_that_carried_it(self, session, corpus): + in_beat, _ = corpus + add_article( + session, in_beat, "NIST publishes vulnerability guidance", datetime(2026, 8, 25) + ) + report = run(session, [probe(terms=("NIST",), any_of=(("vulnerabilit*",),))]) + + (result,) = report.results + assert result.status == STATUS_MATCHED + assert result.match_count == 1 + assert result.matches[0].feed_name == FEED_NAME + assert result.matches[0].title == "NIST publishes vulnerability guidance" + + def test_articles_in_the_window_that_do_not_match_are_a_failure(self, session, corpus): + in_beat, _ = corpus + add_article(session, in_beat, "An unrelated procurement notice", datetime(2026, 8, 25)) + (result,) = run(session, [probe()]).results + assert result.status == STATUS_UNMATCHED + assert result.articles_in_window == 1 + assert result.missing_terms == ("NIST",) + + def test_a_match_outside_the_window_does_not_count(self, session, corpus): + in_beat, _ = corpus + add_article(session, in_beat, "NIST publishes vulnerability guidance", datetime(2026, 6, 1)) + add_article(session, in_beat, "Something else entirely", datetime(2026, 8, 25)) + (result,) = run(session, [probe()]).results + assert result.status == STATUS_UNMATCHED + + def test_a_match_on_another_beats_feed_does_not_count_but_is_reported(self, session, corpus): + in_beat, outside = corpus + add_article(session, in_beat, "An unrelated procurement notice", datetime(2026, 8, 25)) + add_article( + session, outside, "NIST publishes vulnerability guidance", datetime(2026, 8, 25) + ) + (result,) = run(session, [probe()]).results + + assert result.status == STATUS_UNMATCHED + assert result.elsewhere_count == 1 + assert result.elsewhere[0].feed_name == OTHER_NAME + assert result.elsewhere[0].in_beat_sources is False + + def test_no_one_carried_it_is_distinguishable_from_someone_else_did(self, session, corpus): + in_beat, _ = corpus + add_article(session, in_beat, "An unrelated procurement notice", datetime(2026, 8, 25)) + (result,) = run(session, [probe()]).results + assert result.status == STATUS_UNMATCHED + assert result.elsewhere == () + + def test_the_body_is_searched_not_only_the_title(self, session, corpus): + in_beat, _ = corpus + add_article( + session, + in_beat, + "This week in federal cyber", + datetime(2026, 8, 25), + body="Elsewhere, NIST said it was rethinking its vulnerability analysis role.", + ) + (result,) = run(session, [probe(any_of=(("vulnerabilit*",),))]).results + assert result.status == STATUS_MATCHED + + def test_an_undated_article_can_never_satisfy_a_probe(self, session, corpus): + """ + An article with no published_date cannot be placed in any window, so it + must not be able to turn an inconclusive probe into a pass or a failure. + """ + in_beat, _ = corpus + add_article(session, in_beat, "NIST vulnerability guidance", None) + (result,) = run(session, [probe()]).results + assert result.status == STATUS_INCONCLUSIVE + + +class TestInconclusive: + def test_a_probe_predating_the_corpus_is_inconclusive(self, session, corpus): + in_beat, _ = corpus + add_article(session, in_beat, "Recent news", datetime(2026, 8, 25)) + (result,) = run(session, [probe(date=date(2021, 11, 3))]).results + assert result.status == STATUS_INCONCLUSIVE + assert "predates the corpus" in result.reason + + def test_a_probe_the_corpus_has_not_reached_is_inconclusive(self, session, corpus): + in_beat, _ = corpus + add_article(session, in_beat, "Old news", datetime(2026, 1, 5)) + (result,) = run(session, [probe(date=date(2026, 8, 24))]).results + assert result.status == STATUS_INCONCLUSIVE + assert "has not reached the event yet" in result.reason + + def test_a_gap_in_the_middle_is_inconclusive_and_says_so(self, session, corpus): + in_beat, _ = corpus + add_article(session, in_beat, "Before", datetime(2026, 1, 5)) + add_article(session, in_beat, "After", datetime(2026, 12, 5)) + (result,) = run(session, [probe(date=date(2026, 8, 24))]).results + assert result.status == STATUS_INCONCLUSIVE + assert "gap in the corpus" in result.reason + + def test_a_beat_whose_feeds_have_no_rows_is_inconclusive_not_failed(self, session, corpus): + _, outside = corpus + add_article(session, outside, "NIST vulnerability guidance", datetime(2026, 8, 25)) + (result,) = run(session, [probe()]).results + assert result.status == STATUS_INCONCLUSIVE + assert not result.conclusive + + +class TestInconclusiveDoesNotShrinkTheProbeSet: + """ + The landmine: a probe set that quietly decays to the events still in + retention is a green light that means nothing. The counts must always sum + back to the number of probes the beat declared. + """ + + @pytest.fixture + def mixed(self, session, corpus): + in_beat, _ = corpus + add_article( + session, in_beat, "NIST publishes vulnerability guidance", datetime(2026, 8, 25) + ) + add_article(session, in_beat, "An unrelated notice", datetime(2026, 8, 26)) + return run( + session, + [ + probe(what="matches"), + probe(what="misses", terms=("FedRAMP",), any_of=(("director",),)), + probe(what="too old", date=date(2021, 11, 3)), + probe(what="also too old", date=date(2020, 1, 1)), + ], + ) + + def test_every_declared_probe_appears_in_the_results(self, mixed): + assert mixed.total == 4 + assert [r.probe.what for r in mixed.results] == [ + "matches", + "misses", + "too old", + "also too old", + ] + + def test_the_three_counts_sum_to_the_number_declared(self, mixed): + assert len(mixed.matched) + len(mixed.unmatched) + len(mixed.inconclusive) == mixed.total + assert (len(mixed.matched), len(mixed.unmatched), len(mixed.inconclusive)) == (1, 1, 2) + + def test_conclusive_is_a_smaller_set_and_says_so(self, mixed): + assert len(mixed.conclusive) == 2 + assert len(mixed.conclusive) < mixed.total + + +class TestItNeverWrites: + """ + The corpus is shared. A question about it must not alter it. + """ + + def test_running_probes_leaves_the_row_counts_alone(self, session, corpus): + in_beat, _ = corpus + add_article(session, in_beat, "NIST vulnerability guidance", datetime(2026, 8, 25)) + before = (session.query(Article).count(), session.query(RSSFeed).count()) + + run(session, [probe(), probe(date=date(2021, 1, 1))]) + + session.expire_all() + assert (session.query(Article).count(), session.query(RSSFeed).count()) == before + + def test_the_session_is_never_flushed_dirty(self, session, corpus): + """Nothing is added to the session, so a rollback loses nothing.""" + in_beat, _ = corpus + add_article(session, in_beat, "NIST vulnerability guidance", datetime(2026, 8, 25)) + run(session, [probe()]) + assert not session.new + assert not session.deleted + assert not session.dirty + + +class TestFeedResolution: + def test_feed_urls_default_to_the_beats_own_resolution(self, session, corpus, monkeypatch): + """ + With no explicit ``feed_urls`` the beat resolves its own feeds through + FeedMatcher, so the command reflects the beat file rather than a caller's + idea of it. + """ + in_beat, _ = corpus + add_article( + session, in_beat, "NIST publishes vulnerability guidance", datetime(2026, 8, 25) + ) + + class FakeMatcher: + all_feeds = [ + Feed( + name=FEED_NAME, + url=FEED_URL, + scope=[], + geo_tags=[], + domain_tags=["regulatory"], + specialty_tags=[], + relevance_score=0.9, + source_file="fake.json", + ) + ] + + monkeypatch.setattr("src.config.beats.FeedMatcher", FakeMatcher) + beat, _urls = beat_with([probe(any_of=(("vulnerabilit*",),))]) + report = run_coverage_probes(session, beat) + + assert report.feed_names == (FEED_NAME,) + assert report.results[0].status == STATUS_MATCHED From 48104698efb5ace6ae23a9b4a809e52ed240edde Mon Sep 17 00:00:00 2001 From: sam-aydlette Date: Thu, 27 Aug 2026 14:20:35 -0400 Subject: [PATCH 3/4] tests: isolate the word-boundary anchor from the acronym case rule TestSubstringMatchingWouldLie carried this task's central claim -- that word-boundary matching is load-bearing -- but could not actually check it. Every collision in NIST_COLLISIONS is lowercase, and a shouted term like NIST compiles case-sensitively, so case-sensitivity rejected those headlines before the boundary anchors were ever consulted. Measured by stripping the anchors from term_pattern in place, leaving the case rule untouched: that class stays fully green, 12 passed. That is not academic. The shipped beat's probes use shouted terms -- OPM and CISA -- so a regression that dropped the anchoring would have left exactly the production term type unguarded. The new class holds case constant instead: term and headline are both all-caps, so the boundary anchor is the only thing left that can reject the match. OPM inside DOPMA and TOPMOST, BOD inside BODY, OMB inside BOMBING, CISA inside PRECISA. Each case pairs the negative assertion with a check that the probe's *other* required term is present, so a headline that merely lacked everything could not satisfy it, and with a positive control asserting the same probe still matches the term standing alone. Verified by the same method: with the anchors stripped from the production regex, 13 of the 17 fail and the 4 that pass are the positive controls, which should still match. No change to the matcher, the schema, the beat config, the CLI, or any existing test. --- tests/context/test_coverage_probe.py | 82 ++++++++++++++++++++++++++++ 1 file changed, 82 insertions(+) diff --git a/tests/context/test_coverage_probe.py b/tests/context/test_coverage_probe.py index 3327b99..1ef5d4c 100644 --- a/tests/context/test_coverage_probe.py +++ b/tests/context/test_coverage_probe.py @@ -123,6 +123,88 @@ def test_an_acronym_only_matches_when_it_is_shouted(self): assert not compiled.matches("A bod issued a statement") +# All-caps collisions, for holding case constant while the boundary is tested. +# Each headline embeds a shouted probe term inside a longer shouted word, so +# case-sensitivity cannot be the thing that rejects it. The words are the ones +# the hazard is actually made of: DOPMA is the Defense Officer Personnel +# Management Act, PRECISA is ordinary Portuguese, and an all-caps headline is a +# normal thing for a wire feed to emit. +SHOUTED_COLLISIONS = [ + # (probe term, second required term that IS present, headline) + ("OPM", "REPORT", "COMPTROLLER REPORT ON DOPMA REFORM ADVANCES"), + ("OPM", "REPORT", "AGENCY REPORT NAMES TOPMOST PRIORITIES"), + ("BOD", "CISA", "BODY CAMERA RULE ADVANCES AT CISA"), + ("BOD", "CISA", "CISA ISSUES BODY CAMERA GUIDANCE"), + ("OMB", "FBI", "FBI CONFIRMS BOMBING SUSPECT IN CUSTODY"), + ("CISA", "RELATORIO", "GOVERNO PRECISA DE NOVO RELATORIO SOBRE SEGURANCA"), +] + + +class TestBoundariesAreLoadBearingForShoutedTermsToo: + """ + The boundary isolated from the case rule. + + Every collision in :data:`NIST_COLLISIONS` is lowercase, so a shouted term + like ``NIST`` is already rejected by case-sensitivity before the boundary + anchors are consulted -- those cases would stay green if the anchors were + deleted outright. That is not a hypothetical gap: the shipped beat's probes + are shouted (``OPM``, ``CISA``), so it is exactly the term type in + production whose anchoring would go unguarded. + + Here the headline and the term are both ALL CAPS. Case is held constant, so + the boundary anchor is the only thing left that can reject the match. Each + case was checked against the production regex with the anchors stripped and + case matching unchanged: all six then match, which is what makes these + assertions load-bearing rather than incidental. + """ + + @pytest.mark.parametrize(("term", "other", "title"), SHOUTED_COLLISIONS) + def test_a_shouted_term_inside_a_shouted_word_is_not_a_match(self, term, other, title): + compiled = compile_probe(probe(terms=(term, other), any_of=())) + assert not compiled.matches(title) + + @pytest.mark.parametrize(("term", "other", "title"), SHOUTED_COLLISIONS) + def test_the_second_required_term_is_present_so_only_the_collision_fails( + self, term, other, title + ): + """ + Pins *why* the case above fails. Without this, a headline that simply + lacked both terms would satisfy the assertion and prove nothing about + the boundary. + """ + compiled = compile_probe(probe(terms=(term, other), any_of=())) + absent, _ = compiled.missing(title) + assert absent == (term,) + + @pytest.mark.parametrize( + ("term", "other", "title"), + [ + ("OPM", "REPORT", "OPM REPORT ON DOPMA REFORM ADVANCES"), + ("BOD", "CISA", "CISA ISSUES BOD 22-01"), + ("OMB", "FBI", "FBI CITES OMB MEMO M-26-01"), + ("CISA", "RELATORIO", "CISA PUBLICA NOVO RELATORIO SOBRE SEGURANCA"), + ], + ) + def test_the_same_probe_still_matches_the_term_standing_alone(self, term, other, title): + """ + The other half of the claim: the anchors reject the collision without + rejecting the event. A matcher that failed both would pass the tests + above and be useless. + """ + compiled = compile_probe(probe(terms=(term, other), any_of=())) + assert compiled.matches(title) + + def test_the_right_hand_anchor_is_exercised_at_position_zero(self): + """ + ``BODY CAMERA...`` puts the collision at offset 0, where there is no + preceding character for the left-hand lookbehind to reject. Only the + right-hand anchor can refuse it. + """ + compiled = compile_probe(probe(terms=("BOD",), any_of=())) + assert not compiled.matches("BODY CAMERA RULE ADVANCES") + assert compiled.matches("BOD 22-01 ADVANCES") + + class TestStems: def test_a_stem_must_be_marked(self): assert not compile_probe(probe(terms=("reinstat", "x"), any_of=())).matches( From d2e331cf0ba4be9e6eee6f23478eb607bb141825 Mon Sep 17 00:00:00 2001 From: sam-aydlette Date: Thu, 27 Aug 2026 14:23:05 -0400 Subject: [PATCH 4/4] test: correct a docstring that claimed the boundary was doing the work TestSubstringMatchingWouldLie said 'The boundary, not the two-level shape, is what rejects these.' That is false for its own cases: every term in it is shouted, so the compiled pattern is case-sensitive and the lowercase collisions are refused before the anchors are consulted. Deleting LEFT_BOUNDARY and RIGHT_BOUNDARY from entity_matcher leaves all 12 passing. The IC that found this correctly declined to edit an existing test outside its scope and flagged it instead. Correcting it here rather than leaving a demonstrably false claim next to the class that disproves it. --- tests/context/test_coverage_probe.py | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/tests/context/test_coverage_probe.py b/tests/context/test_coverage_probe.py index 1ef5d4c..ea827f6 100644 --- a/tests/context/test_coverage_probe.py +++ b/tests/context/test_coverage_probe.py @@ -112,7 +112,7 @@ def test_probe_matching_reports_only_the_real_ones(self): @pytest.mark.parametrize("title", NIST_COLLISIONS) def test_no_collision_matches_even_the_bare_term(self, title): - """The boundary, not the two-level shape, is what rejects these.""" + """For these lowercase collisions it is the case rule, not the boundary, that rejects them: every term here is shouted, so the compiled pattern is case-sensitive and never reaches the anchors. Verified 2026-08-27 by deleting LEFT_BOUNDARY/RIGHT_BOUNDARY from entity_matcher and re-running this class -- all 12 still passed. See TestBoundariesAreLoadBearingForShoutedTermsToo, which holds case constant so only the anchor can refuse the match.""" compiled = compile_probe(probe(terms=("NIST", "the"), any_of=())) assert not compiled.matches(title)