diff --git a/.gitignore b/.gitignore index 689343b..871071b 100644 --- a/.gitignore +++ b/.gitignore @@ -23,6 +23,11 @@ config/api_sources.json # Philosophical documentation (not for version control) docs/philosophical_docs/ +# Tier 1's coverage-gap report (backlog task 015, 2026-08-31). Derived: it is +# rewritten from the observations on every `insightweaver route` run, and it +# quotes the operator's corpus. Task 021 reads it from disk, not from git. +data/routing/ + # Database files *.db *.db-journal diff --git a/Makefile b/Makefile index 22a1473..edff535 100644 --- a/Makefile +++ b/Makefile @@ -1,4 +1,4 @@ -.PHONY: help install install-dev test lint format fmt typecheck check pre-commit clean clean-all coverage db-add-watches db-add-observations db-drop-briefing update-deps +.PHONY: help install install-dev test lint format fmt typecheck check pre-commit clean clean-all coverage db-add-watches db-add-observations db-add-routes db-drop-briefing update-deps # Tool resolution (added 2026-08-24). # Before this, every recipe called bare `pytest` / `ruff` / `mypy`, which only @@ -46,6 +46,7 @@ help: @echo "Database:" @echo " make db-add-watches Create the watches table (additive, safe)" @echo " make db-add-observations Create observations and evidence (additive, safe)" + @echo " make db-add-routes Create the route_candidates table (additive, safe)" @echo " make db-drop-briefing Drop the deleted briefing product's tables" @echo " (DESTRUCTIVE; captures to a dump first)" @echo "" @@ -127,6 +128,13 @@ db-add-watches: db-add-observations: $(PYTHON) -m src.database.migrations.add_observations_and_evidence +# Additive. Creates Tier 1's output table. The unique constraint on +# (observation_hash, watch_id) it carries is what makes routing idempotent, so +# the table is created from the model rather than from hand-written DDL. +# (2026-08-31, backlog task 015.) +db-add-routes: + $(PYTHON) -m src.database.migrations.add_route_candidates + # Deliberately does NOT pass --confirm. The migration refuses without it, so # running this target prints what it would destroy and stops; the operator types # the real command themselves. A Makefile target that drops tables on `make` is diff --git a/README.md b/README.md index 345e5fb..5e3f8d0 100644 --- a/README.md +++ b/README.md @@ -584,6 +584,63 @@ Create the tables with `make db-add-observations` (additive; it does not touch --- +## Tier 1: deterministic routing + +Added 2026-08-31 (`backlog/015`). Adjudication is the only tier that calls a +model, and it costs money per item. Tier 1 is what decides which items it ever +sees, so **notification volume scales with the number of live watches rather +than with news volume** -- and that is a property of the code in `src/routing/`, +not an aspiration. + +**Triggers compile; they are not interpreted and not read by a model.** A +watch's `triggers` is a list of clauses over `terms`, `entities` and `sources`. +Within a clause every populated field must match (AND) and any value in a field +will do (OR); across clauses, any clause firing fires the watch. That compiles +directly into two regex alternations and a set-membership test. Nothing in +`src/routing/` imports an Anthropic client, and the test that says so removes it +from the interpreter entirely and drives the whole path +(`tests/routing/test_no_model.py`). + +**Word boundaries are load-bearing.** Every pattern comes from +`src/matching/entity_matcher.py`'s `compile_terms`, which owns the anchors and +the shouted-acronym case rule -- there is deliberately no second matcher. The +scale being defended, measured on this repository's own 55,249-article corpus: +`nist` appears as a substring in 5,364 titles and at a word boundary in 73; +`mail` is 1,842 against 339. + +**Routing is idempotent.** A match becomes a `route_candidates` row, unique on +`(observation_hash, watch_id)`; routing the same observation twice produces one +link. The row also records which clause fired, which is the difference between +"this watch routed 400 observations" and "clause 2 of this watch routed 400". + +```bash +# Per watch, how many of the last 500 observations would route -- plus the +# unrouted count and its clusters. Writes no route_candidates rows. +insightweaver route --dry-run + +# The same, over every stored observation, recording the links. +insightweaver route --limit 0 +``` + +**Read the unrouted number first.** A high one is the healthy state. A low one +means a trigger is too loose, and a trigger that is too loose does not fail -- +it bills. The unrouted observations are clustered by their stored MinHash +signatures and written to `data/routing/unrouted_clusters.json` alongside a +document-frequency histogram of their salient terms. That file is the +coverage-gap signal: the only place a *missing* watch is visible before a +staleness alert fires. `backlog/021` reads it. + +**The ceiling test.** `tests/routing/test_ceiling.py` asserts that 1,000 +observations against 5 watches route fewer than 20. The number was measured, not +guessed: the corpus is 1,000 real articles sampled from the pre-rewrite archive +and stratified by feed category, the measured baseline is 5 routed / 5 links, +and a clause whose AND became an OR routes 33. The test's docstring records the +measurement and is explicit about what it cannot see. + +Create the table with `make db-add-routes` (additive). + +--- + ## Requirements - Python 3.10+ diff --git a/backlog/015-tier1-deterministic-routing.md b/backlog/015-tier1-deterministic-routing.md index 7e4a2f1..160b8c8 100644 --- a/backlog/015-tier1-deterministic-routing.md +++ b/backlog/015-tier1-deterministic-routing.md @@ -15,3 +15,45 @@ rather than news volume -- is actually enforced. Tiers 2 through 4 can only pres in production metrics is one that regresses between releases and is noticed at the invoice. A test asserting that N observations against M watches routes fewer than K makes the property a gate. Pick K from a measured baseline, not from intuition, and record how it was measured. + +--- + +## Done 2026-08-31 (branch `task/015-routing`) + +Built `src/routing/` (predicate, router, salience, gaps), `src/cli/route.py`, +the `route_candidates` table and its migration. `make check` exits 0 and +`env -u ANTHROPIC_API_KEY pytest tests/ -q` exits 0 (730 passed). + +**K = 20, measured, not guessed.** The ceiling fixture is 1,000 *real* articles +lifted from the 55,249-row pre-rewrite archive through a read-only +(`immutable=1`) SQLite handle, stratified 400/600 by feed category (not by +trigger word -- that would have been circular) and stored through the real +`store_observation` path. Measured baseline: 5 of 1,000 route, 5 links, fan-out +0.005. A clause whose AND became an OR routes 33; dropping the term/entity +constraints routes 1,000. K = 20 sits between. Full derivation, including what +the ceiling does *not* gate, is in `tests/routing/test_ceiling.py`'s docstring. +Nothing was written to any database that outlives the measurement; this is not a +backfill and must not be read as one. + +**Word boundaries were verified by deletion, not by assertion.** With +`LEFT_BOUNDARY`/`RIGHT_BOUNDARY` emptied, `TestWordBoundariesAreLoadBearing` +goes 15 failed / 3 passed; restored, 18 passed. Every case in that class uses a +non-shouted term (`nist`, `mail`, `ai`, `rev 5`), so the case rule cannot reject +them first -- which is the trap task 010 fell into. + +### Two things for you, neither of them blocking + +1. **`sources:` is matched by exact equality, and that is a live question.** A + clause saying `sources: [Federal Register]` matches a feed *named* exactly + that. The corpus has three: "Federal Register", "Federal Register - Public + Inspection", "Federal Register - Documents API", and only the last carries + articles -- so `config/watches.example.yaml`'s own source clause matches + nothing here. Equality was chosen because it can only under-route, and + under-routing surfaces in the unrouted clusters rather than on an invoice. + If you meant the family of feeds, say so and it becomes a prefix or a + category match. +2. **Trigger `entities:` currently match only the surface form you wrote.** The + alias registry that used to supply `CoverageEntity.aliases` died with the + beat files in task 012 and nothing replaced it. So `entities: [FedRAMP PMO]` + does not also match "FedRAMP Program Management Office". The one place that + changes is `src.routing.predicate._entity_terms`. diff --git a/src/cli/app.py b/src/cli/app.py index baccb7f..abd7c2f 100644 --- a/src/cli/app.py +++ b/src/cli/app.py @@ -20,6 +20,7 @@ from .colors import accent, header, muted from .output import set_debug_mode from .replay import replay_command +from .route import route_command from .sources import sources_command from .watch import watch_command @@ -33,6 +34,7 @@ "sources": sources_command, "watch": watch_command, "replay": replay_command, + "route": route_command, } @@ -40,7 +42,7 @@ def print_command_refresher(): """Print a short refresher of available commands.""" click.echo( f"\n{header('Commands:')} {accent('sources')} | {accent('watch')} | " - f"{accent('replay')} | help | exit\n" + f"{accent('route')} | {accent('replay')} | help | exit\n" ) @@ -51,6 +53,9 @@ def print_help(): click.echo( f" {accent('watch')} - Pre-registered watches and the decisions they serve" ) + click.echo( + f" {accent('route')} - Match Observations to Watches, deterministically" + ) click.echo( f" {accent('replay')} - Rebuild Evidence from Observations and diff it" ) @@ -72,6 +77,11 @@ def print_help(): muted(" Watches are only ever created from those files -- there is no 'watch add'.") ) click.echo() + click.echo(header("Route command:")) + click.echo(f" {accent('route --dry-run')} - Per watch, how many of the last N route") + click.echo(muted(" Reports the unrouted count and its clusters -- the coverage-gap signal.")) + click.echo(muted(" No model is involved anywhere in this tier.")) + click.echo() click.echo(header("Replay command:")) click.echo(f" {accent('replay --prompt-version X')} - Rebuild Evidence for prompt version X") click.echo( diff --git a/src/cli/route.py b/src/cli/route.py new file mode 100644 index 0000000..7f2140f --- /dev/null +++ b/src/cli/route.py @@ -0,0 +1,197 @@ +""" +Route command - which observations are candidate evidence for which watch. + +``--dry-run`` is the reporting mode and it writes no ``route_candidates`` rows. +It answers the question this tier exists to make answerable: of the last N +observations, how many would each watch send to the adjudicator, which clause +of which watch is doing the sending, and -- the part that is easy to leave out +and is the whole coverage-gap signal -- how many matched nothing, and what were +they about. + +**Read the unrouted number first.** A high one is the expected, healthy state: +most of what a feed publishes is not evidence about anything you have +pre-registered a claim on, and if it were, the architecture would not be doing +its job. A *low* one means a trigger is too loose, and a trigger that is too +loose does not fail -- it bills, because the adjudicator is the only tier that +calls a model and it runs on whatever this one hands it. + +The coverage-gap report is written to disk in both modes, because it is a report +derived from the corpus rather than a change to it, and because the mode you +would run to find a gap is the dry run. Its path is printed. Backlog task 021 +reads it. + +Added 2026-08-31 for backlog task 015. +""" + +from datetime import date + +import click + +from ..database.connection import get_db +from ..database.models import Observation, Watch +from ..routing import ( + TriggerCompileError, + compile_watches, + gap_report, + persist, + route, + write_gap_report, +) +from .colors import accent, header, muted, warning + +# Why a default at all: near-duplicate clustering of the unrouted set is +# pairwise, so it is quadratic in the number of observations that matched +# nothing -- which is most of them. 500 is a few days of ingestion and runs in +# about a second. --limit 0 means every stored observation and is the honest +# spelling of "I know what I am asking for". +_DEFAULT_LIMIT = 500 + + +def _clause_lines(compiled, per_clause) -> list[str]: + """One line per trigger clause that fired, so a wide trigger is attributable.""" + lines = [] + for watch in compiled: + for clause in watch.clauses: + count = per_clause.get((watch.watch_id, clause.index), 0) + if count: + lines.append(f" clause {clause.index}: {count:>6} {clause.describe()}") + return lines + + +@click.command(name="route") +@click.option( + "--dry-run", + is_flag=True, + default=False, + help="Report what would route and write no route_candidates rows.", +) +@click.option( + "--limit", + type=int, + default=_DEFAULT_LIMIT, + show_default=True, + help="Consider the last N observations. 0 means all of them.", +) +@click.option( + "--gaps-out", + type=click.Path(dir_okay=False), + default=None, + help="Where to write the unrouted cluster report. Defaults to data/routing/.", +) +@click.option( + "--no-gaps", + is_flag=True, + default=False, + help="Skip the unrouted clustering. Faster, and loses the coverage-gap signal.", +) +def route_command(dry_run, limit, gaps_out, no_gaps): + """Route observations to watches with compiled, deterministic predicates.""" + today = date.today() + + with get_db() as session: + stored = session.query(Observation).count() + if stored == 0: + click.echo( + warning( + "No observations are stored, so there is nothing to route. Tier 1 reads " + "'observations' and never 'articles'; the 55,249 legacy rows are the " + "pre-rewrite archive. See src/database/models.py." + ) + ) + return + + watch_rows = session.query(Watch).all() + if not watch_rows: + click.echo( + warning( + "No watches are stored. Routing with no watches routes nothing, which is " + "correct and useless. Run 'watch sync' first." + ) + ) + return + + try: + report = route(session, limit=limit or None, today=today) + compiled = compile_watches( + [row for row in watch_rows if row.expires is None or row.expires >= today] + ) + except TriggerCompileError as exc: + raise click.ClickException(str(exc)) + + click.echo(header("ROUTE" + (" (dry run)" if dry_run else ""))) + click.echo( + muted( + f"{report.considered} of {stored} stored observation(s), newest first, " + f"against {len(report.watch_ids)} live watch(es)." + ) + ) + if report.expired_watch_ids: + click.echo( + muted(f" skipped {len(report.expired_watch_ids)} expired: ") + + muted(", ".join(report.expired_watch_ids)) + ) + click.echo("=" * 80) + + for watch_id in report.watch_ids: + count = report.per_watch[watch_id] + share = count / report.considered if report.considered else 0.0 + click.echo(f" {accent(watch_id):<48} {count:>6} ({share:6.2%})") + click.echo() + for line in _clause_lines(compiled, report.per_clause): + click.echo(muted(line)) + + click.echo() + click.echo( + f" {'routed':<48} {report.routed_count:>6} " + f"({report.routed_count / report.considered:6.2%} of considered)" + ) + click.echo( + f" {'unrouted':<48} {report.unrouted_count:>6} " + f"({report.unrouted_count / report.considered:6.2%} of considered)" + ) + click.echo( + muted( + f" {len(report.links)} link(s), fan-out {report.fan_out:.2f} " + f"adjudications per observation considered." + ) + ) + + if not no_gaps: + payload = gap_report(session, report) + path = write_gap_report(payload, gaps_out) + clusters = payload["clusters"] + multi = [c for c in clusters if c["size"] > 1] + click.echo() + click.echo(header("UNROUTED CLUSTERS")) + click.echo( + muted( + f"{len(clusters)} cluster(s), {len(multi)} with more than one member. " + f"A large one is a story the environment thought mattered and no watch " + f"can see." + ) + ) + for cluster in clusters[:10]: + terms = ", ".join(term for term, _ in cluster["terms"][:5]) + click.echo(f" {cluster['size']:>4} {cluster['representative_title'][:60]}") + click.echo(muted(f" {terms}")) + if payload["gap_terms"]: + top = ", ".join(f"{t} ({n})" for t, n in payload["gap_terms"][:12]) + click.echo() + click.echo(muted(f" gap terms: {top}")) + click.echo() + click.echo(muted(f" written to {path}")) + + click.echo() + if dry_run: + click.echo( + muted("Nothing was written to route_candidates. Re-run without ") + + accent("--dry-run") + + muted(" to record these links.") + ) + return + + written = persist(session, report) + click.echo( + f"routed: {written['inserted']} link(s) inserted, " + f"{written['already_linked']} already present" + ) diff --git a/src/database/migrations/add_route_candidates.py b/src/database/migrations/add_route_candidates.py new file mode 100644 index 0000000..eac0529 --- /dev/null +++ b/src/database/migrations/add_route_candidates.py @@ -0,0 +1,89 @@ +""" +Migration: add the ``route_candidates`` table. + +Additive. It creates one table and touches nothing that exists -- in particular +it does not read, rewrite or migrate ``articles``, and it does not touch +``observations``, whose rows are immutable. + +The DDL is not written out here. The table is created from its model, as +``add_watches_table`` and ``add_observations_and_evidence`` do, so the migration +and the models cannot drift. The unique constraint on +``(observation_hash, watch_id)`` is what makes routing idempotent, and a +hand-written CREATE TABLE that forgot it would produce a table that looks right +and silently doubles every watch's candidate count on the second run. + +``downgrade()`` requires ``--confirm``. What it destroys is derived -- routing +is deterministic, so re-running it rebuilds every row from the same observations +-- but it also destroys the record of *when* each candidate was routed, which is +not. + +Added 2026-08-31 for backlog task 015. +""" + +from __future__ import annotations + +import argparse +import sys + +from sqlalchemy import inspect +from sqlalchemy.engine import Engine + +from src.database.connection import engine +from src.database.models import RouteCandidate + +__all__ = ["TABLE", "downgrade", "upgrade"] + +TABLE = "route_candidates" + +_CONFIRM_HELP = ( + "Dropping route_candidates discards every Tier 1 routing decision and the\n" + "time each was made. The decisions are recomputable -- routing is\n" + "deterministic -- but the routed_at timestamps are not.\n" + "Re-run with --confirm if that is what you want:\n\n" + " python -m src.database.migrations.add_route_candidates --down --confirm\n" +) + + +def upgrade(target: Engine | None = None) -> list[str]: + """Create the table if it is absent. Returns the names created.""" + target = target or engine + if inspect(target).has_table(TABLE): + print(f"{TABLE} already exists; nothing to do.") + return [] + RouteCandidate.__table__.create(bind=target) + print(f"Created {TABLE} ({len(RouteCandidate.__table__.columns)} columns).") + return [TABLE] + + +def downgrade(target: Engine | None = None, *, confirmed: bool = False) -> list[str]: + """Drop the table. Requires ``confirmed`` -- this loses rows.""" + if not confirmed: + raise SystemExit(_CONFIRM_HELP) + target = target or engine + if not inspect(target).has_table(TABLE): + print(f"{TABLE} does not exist; nothing to do.") + return [] + RouteCandidate.__table__.drop(bind=target) + print(f"Dropped {TABLE}.") + return [TABLE] + + +def main(argv: list[str] | None = None) -> int: + parser = argparse.ArgumentParser( + prog="add_route_candidates", + description=__doc__, + formatter_class=argparse.RawDescriptionHelpFormatter, + ) + parser.add_argument("--down", action="store_true", help="drop the table again") + parser.add_argument("--confirm", action="store_true", help="required by --down") + args = parser.parse_args(argv) + + if args.down: + downgrade(confirmed=args.confirm) + return 0 + upgrade() + return 0 + + +if __name__ == "__main__": + sys.exit(main()) diff --git a/src/database/models.py b/src/database/models.py index fd2294b..97cf84f 100644 --- a/src/database/models.py +++ b/src/database/models.py @@ -15,7 +15,9 @@ 013. Its CHECK constraints are not decoration -- see the class docstring. ``observations`` and ``evidence`` are the second and third, added 2026-08-31 by -backlog task 014. +backlog task 014. ``route_candidates`` is the fourth, added 2026-08-31 by +backlog task 015: it holds Tier 1's output, which is candidacy rather than +evidence. **The rule for ``articles`` versus ``observations``, decided by task 014 and written here because this is the file both of them live in.** They coexist, with @@ -382,3 +384,56 @@ class Evidence(Base): Index("idx_evidence_prompt_version", "prompt_version"), Index("idx_evidence_watch", "watch_id"), ) + + +class RouteCandidate(Base): + """ + One observation that a Watch's compiled triggers selected as *candidate* evidence. + + Tier 1's output, and nothing more than that. A row here says "this + observation matched this watch's triggers", which is a claim about regexes + and a source allowlist -- not a claim that the observation is evidence, and + not a direction. Adjudication produces :class:`Evidence`; this table is what + adjudication is allowed to read. Keeping them separate is what makes the + cost claim checkable: the number of rows in this table is the number of + observations the one stochastic tier will ever see. + + **The unique constraint is the idempotency.** Routing the same observation + twice produces one link, because ``(observation_hash, watch_id)`` is unique + and :func:`src.routing.router.persist` skips what is already there. Tier 1 + is deterministic, so re-running it over an unchanged corpus is a no-op -- + and if it ever is not, the constraint says so with an IntegrityError rather + than doubling a watch's candidate count. + + ``clause_index`` records *which* trigger clause fired. It is the difference + between "this watch is routing 400 observations" and "clause 2 of this watch + is routing 400 observations", which is the difference between knowing a + trigger is too loose and knowing which line of the file to edit. It is + excluded from the unique key on purpose: an observation matching two clauses + of the same watch is still one candidate, and the first clause in file order + is the one recorded. + + There is no ``routed_by`` or model column, because there is no model. The + test that proves it is + tests/routing/test_no_model.py::TestRoutingNeedsNoAnthropicClient. + + Added 2026-08-31 for backlog task 015. + """ + + __tablename__ = "route_candidates" + + id = Column(Integer, primary_key=True) + + observation_hash = Column(String(80), ForeignKey("observations.content_hash"), nullable=False) + watch_id = Column(String(100), ForeignKey("watches.id"), nullable=False) + + clause_index = Column(Integer, nullable=False) + + routed_at = Column(DateTime, default=utcnow, nullable=False) + + __table_args__ = ( + UniqueConstraint("observation_hash", "watch_id", name="_route_observation_watch_uc"), + CheckConstraint("clause_index >= 0", name="ck_route_clause_index"), + Index("idx_route_watch", "watch_id"), + Index("idx_route_observation", "observation_hash"), + ) diff --git a/src/matching/entity_matcher.py b/src/matching/entity_matcher.py index 8bced2b..ba22e73 100644 --- a/src/matching/entity_matcher.py +++ b/src/matching/entity_matcher.py @@ -42,9 +42,11 @@ "RIGHT_BOUNDARY", "CompiledEntity", "compile_entities", + "compile_terms", "count_item_mentions", "is_shouted", "item_text", + "matches_any", "term_pattern", ] @@ -91,6 +93,50 @@ def term_pattern(term: str, right_boundary: bool = True) -> str: return f"{LEFT_BOUNDARY}(?:{body}){tail}" +def compile_terms(terms: Iterable[str]) -> tuple[re.Pattern[str], ...]: + """ + A group of surface forms compiled to the fewest regexes that can hold them. + + Shouted and unshouted terms go into separate alternations so each group can + carry the case flag it needs -- ``BOD`` must not be matched by ``body`` and + ``FedRAMP PMO`` must be matched by ``fedramp pmo``. A term that is only + whitespace is dropped rather than compiled into a pattern that matches + everywhere. + + Extracted from :func:`compile_entities` on 2026-08-31 (backlog task 015) so + that Tier 1 routing compiles a trigger clause's ``terms`` and ``entities`` + through *this* function rather than through a second boundary + implementation. The repository has already been bitten by substring + matching: ``nist`` is a substring of 5,364 article titles in the corpus and + a word-boundary match in 73. One matcher, one set of anchors. + """ + shouted: list[str] = [] + relaxed: list[str] = [] + for term in terms: + if not term.strip(): + continue + (shouted if is_shouted(term) else relaxed).append(term_pattern(term)) + + patterns: list[re.Pattern[str]] = [] + if shouted: + patterns.append(re.compile("|".join(shouted))) + if relaxed: + patterns.append(re.compile("|".join(relaxed), re.IGNORECASE)) + return tuple(patterns) + + +def matches_any(patterns: Iterable[re.Pattern[str]], text: str) -> bool: + """ + Whether any of ``patterns`` hits ``text``. Presence, not frequency. + + An empty ``patterns`` is False: nothing was looked for, so nothing was + found. Callers that mean "unconstrained" must not reach here at all -- + see :class:`src.routing.predicate.CompiledClause`, which refuses to build a + clause whose declared field compiled to no patterns. + """ + return any(pattern.search(text) for pattern in patterns) + + @dataclass(frozen=True) class CompiledEntity: """One coverage entity with its surface forms compiled to regexes.""" @@ -110,33 +156,19 @@ def mentioned_in(self, text: str) -> bool: item that mentioned CISA, and counting the repetitions would make a verbose outlet look like institutional activity. """ - return any(pattern.search(text) for pattern in self.patterns) + return matches_any(self.patterns, text) def compile_entities(entities: Iterable[CoverageEntity]) -> list[CompiledEntity]: """ Compile every entity's surface forms once, for reuse across all items. - Shouted and unshouted terms are grouped into separate alternations so each - group can carry the case flag it needs; a term that is only whitespace is - dropped rather than compiled into a pattern that matches everywhere. + The grouping and case rules live in :func:`compile_terms`; this function is + the entity-shaped wrapper around them. """ - compiled: list[CompiledEntity] = [] - for entity in entities: - shouted: list[str] = [] - relaxed: list[str] = [] - for term in entity.terms: - if not term.strip(): - continue - (shouted if is_shouted(term) else relaxed).append(term_pattern(term)) - - patterns: list[re.Pattern[str]] = [] - if shouted: - patterns.append(re.compile("|".join(shouted))) - if relaxed: - patterns.append(re.compile("|".join(relaxed), re.IGNORECASE)) - compiled.append(CompiledEntity(entity=entity, patterns=tuple(patterns))) - return compiled + return [ + CompiledEntity(entity=entity, patterns=compile_terms(entity.terms)) for entity in entities + ] def item_text(item: Mapping[str, Any]) -> str: diff --git a/src/routing/__init__.py b/src/routing/__init__.py new file mode 100644 index 0000000..a367725 --- /dev/null +++ b/src/routing/__init__.py @@ -0,0 +1,57 @@ +""" +Tier 1: deterministic routing from Observations to Watches. + +Nothing in this package calls a model, and nothing in it may. Tier 1 decides +*candidacy* -- whether an observation is worth the adjudicator's attention -- +using compiled word-boundary regexes and a source allowlist, and that is the +whole of it. The tier that reasons is Tier 2, it is the only one that costs +money per item, and it sees exactly what this package hands it. + +Three modules: + +* :mod:`predicate` compiles a watch's ``triggers`` into a predicate. +* :mod:`router` runs the predicates over stored observations and records the + matches as :class:`~src.database.models.RouteCandidate` rows. +* :mod:`gaps` clusters what matched nothing, which is the coverage-gap signal. + +Added 2026-08-31 for backlog task 015. +""" + +from .gaps import ( + GAP_REPORT_SCHEMA, + UnroutedCluster, + cluster_unrouted, + default_gap_report_path, + gap_report, + gap_terms, + write_gap_report, +) +from .predicate import ( + CompiledClause, + CompiledWatch, + TriggerCompileError, + compile_watch, + compile_watches, + source_keys, +) +from .router import RoutedLink, RoutingReport, persist, route + +__all__ = [ + "GAP_REPORT_SCHEMA", + "CompiledClause", + "CompiledWatch", + "RoutedLink", + "RoutingReport", + "TriggerCompileError", + "UnroutedCluster", + "cluster_unrouted", + "compile_watch", + "compile_watches", + "default_gap_report_path", + "gap_report", + "gap_terms", + "persist", + "route", + "source_keys", + "write_gap_report", +] diff --git a/src/routing/gaps.py b/src/routing/gaps.py new file mode 100644 index 0000000..3566418 --- /dev/null +++ b/src/routing/gaps.py @@ -0,0 +1,215 @@ +""" +What the corpus is talking about that no watch can see. + +**The unrouted count is the coverage-gap signal.** It is the only place a +missing sensor becomes visible *before* a staleness alert fires -- a staleness +alert says a watch has gone quiet, which is also what a watch with a broken +trigger looks like, and neither says "there is a whole subject here you never +registered a watch for". Logging that as a bare integer would make it useless: +"988 unrouted" is not actionable, and nobody reads 988 hashes. + +Two layers of grouping, both deterministic, and neither of them a new algorithm: + +* **Near-duplicate clusters.** The stored MinHash signature of every unrouted + observation, grouped by :func:`src.sources.minhash.group_near_duplicates`, + which is the same grouping the rest of the system uses. A cluster of six is + one story carried by six outlets and seen by no watch, which is the strongest + form of the signal: the environment thought it was important and the operator + has no sensor on it. +* **Gap terms.** Document frequency of salient words across all unrouted + observations. This catches the case near-duplicate grouping cannot: a subject + covered by forty *different* stories over a month, no two of them + near-duplicates. Frequency is counted per observation, not per occurrence, so + one verbose article cannot invent a theme. + +**This module proposes nothing.** Proposing watches from these clusters is +backlog task 021, and it is deliberately not here -- invariant 6 says the system +never authors its own watches, and the way that invariant is lost is one +convenient path at a time. What this produces is a report on disk. A human, or +task 021 under a human, reads it. + +Added 2026-08-31 for backlog task 015. +""" + +from __future__ import annotations + +import json +from dataclasses import dataclass +from datetime import datetime +from pathlib import Path +from typing import Any + +from sqlalchemy.orm import Session + +from ..config.settings import settings +from ..database.models import Observation, RSSFeed +from ..sources.minhash import group_near_duplicates +from ..sources.observation import payload_text +from ..utils import utcnow +from .router import RoutingReport +from .salience import gap_terms + +__all__ = [ + "GAP_REPORT_SCHEMA", + "UnroutedCluster", + "cluster_unrouted", + "default_gap_report_path", + "gap_report", + "gap_terms", + "write_gap_report", +] + +# How many hashes go into one IN (...) clause. Well under SQLite's bound +# parameter ceiling on every version this runs on. +_BATCH = 400 + +# Bumped whenever the JSON below changes shape, so task 021 can refuse a report +# it does not understand instead of reading a field that moved. +GAP_REPORT_SCHEMA = "insightweaver.routing.gaps/1" + + +@dataclass(frozen=True) +class UnroutedCluster: + """ + One group of unrouted observations that are near-duplicates of each other. + + ``terms`` labels the cluster with the words its members share, so the file + is readable without opening every payload. + """ + + members: tuple[str, ...] + representative_title: str + sources: tuple[str, ...] + terms: tuple[tuple[str, int], ...] + first_published: str | None + last_published: str | None + + @property + def size(self) -> int: + return len(self.members) + + def as_dict(self) -> dict[str, Any]: + return { + "size": self.size, + "representative_title": self.representative_title, + "members": list(self.members), + "sources": list(self.sources), + "terms": [[term, count] for term, count in self.terms], + "first_published": self.first_published, + "last_published": self.last_published, + } + + +def _iso(value: datetime | None) -> str | None: + return value.isoformat() if value is not None else None + + +def _load(db: Session, hashes: list[str]) -> dict[str, Any]: + """ + The stored rows for ``hashes``, keyed by hash, read in bounded batches. + + Chunked because ``IN (...)`` is a bound-parameter per element and SQLite has + a ceiling on those. A silent truncation at that ceiling would understate the + coverage gap, which is the one direction this report must never err in. + """ + loaded: dict[str, Any] = {} + for start in range(0, len(hashes), _BATCH): + batch = hashes[start : start + _BATCH] + rows = db.query( + Observation.content_hash, + Observation.source_id, + Observation.payload, + Observation.minhash, + Observation.published_date, + ).filter(Observation.content_hash.in_(batch)) + loaded.update({row[0]: row for row in rows}) + return loaded + + +def cluster_unrouted( + db: Session, + hashes: list[str], + *, + threshold: float | None = None, + rows: dict[str, Any] | None = None, +) -> list[UnroutedCluster]: + """ + Group the unrouted observations by near-duplicate similarity, largest first. + + Singletons are included. They are most of the corpus and they are still the + data: task 021 filters, this function reports. Grouping is pairwise, which + is quadratic -- the CLI's ``--limit`` is what bounds it, and the bound is + documented there rather than hidden as a silent truncation here. + """ + if not hashes: + return [] + + by_hash = _load(db, hashes) if rows is None else rows + feeds = {row.id: row.name for row in db.query(RSSFeed.id, RSSFeed.name)} + + groups = group_near_duplicates( + {h: tuple(by_hash[h][3] or ()) for h in by_hash}, + settings.near_duplicate_threshold if threshold is None else threshold, + ) + + clusters: list[UnroutedCluster] = [] + for members in groups: + titles = [str((by_hash[h][2] or {}).get("title") or "") for h in members] + dates = sorted(d for d in (by_hash[h][4] for h in members) if d is not None) + clusters.append( + UnroutedCluster( + members=tuple(members), + representative_title=next((t for t in titles if t), ""), + sources=tuple(sorted({feeds.get(by_hash[h][1]) or "" for h in members} - {""})), + terms=tuple(gap_terms(titles, limit=8)), + first_published=_iso(dates[0]) if dates else None, + last_published=_iso(dates[-1]) if dates else None, + ) + ) + + return sorted(clusters, key=lambda c: (-c.size, c.members[0])) + + +def gap_report( + db: Session, + report: RoutingReport, + *, + threshold: float | None = None, +) -> dict[str, Any]: + """The whole coverage-gap signal as a JSON-serialisable mapping.""" + rows = _load(db, report.unrouted) + clusters = cluster_unrouted(db, report.unrouted, threshold=threshold, rows=rows) + texts = [payload_text(rows[h][2] or {}) for h in report.unrouted if h in rows] + + return { + "schema": GAP_REPORT_SCHEMA, + "generated_at": utcnow().isoformat(), + "observations_considered": report.considered, + "routed_observations": report.routed_count, + "unrouted_observations": report.unrouted_count, + "links": len(report.links), + "watches": list(report.watch_ids), + "expired_watches": list(report.expired_watch_ids), + "per_watch": dict(sorted(report.per_watch.items())), + "clusters": [cluster.as_dict() for cluster in clusters], + "gap_terms": [[term, count] for term, count in gap_terms(texts)], + } + + +def default_gap_report_path() -> Path: + """ + Where the coverage-gap report lands when nothing says otherwise. + + ``data/routing/unrouted_clusters.json``. Task 021 reads this path; it is a + derived report, rewritten on every run, and holds nothing that cannot be + recomputed from the observations. + """ + return Path(settings.data_dir) / "routing" / "unrouted_clusters.json" + + +def write_gap_report(payload: dict[str, Any], path: Path | str | None = None) -> Path: + """Write the report as indented JSON and return where it went.""" + target = Path(path) if path is not None else default_gap_report_path() + target.parent.mkdir(parents=True, exist_ok=True) + target.write_text(json.dumps(payload, indent=2, sort_keys=False) + "\n", encoding="utf-8") + return target diff --git a/src/routing/predicate.py b/src/routing/predicate.py new file mode 100644 index 0000000..d2668bc --- /dev/null +++ b/src/routing/predicate.py @@ -0,0 +1,264 @@ +""" +Compiling a Watch's ``triggers`` into a deterministic predicate. + +**This is Tier 1, and Tier 1 is the cost control.** Every observation the system +ingests is offered to this predicate; the ones it rejects die here and are never +seen by the adjudicator, which is the only tier that calls a model. So the +property the whole architecture rests on -- notification volume scaling with the +number of live watches rather than with news volume -- is a property of the code +in this file. A predicate that is slightly too loose does not raise, does not +log an error and does not fail a test that only checks it matches what it should: +it quietly bills. + +**Shape.** ``triggers`` is a disjunction of conjunctions, guaranteed by +``src.position._validate``: a list of clauses, each constraining some of +``terms`` / ``entities`` / ``sources``. Within a clause every populated field +must match (AND) and any value inside a field will do (OR); across clauses, any +clause firing fires the watch. That is exactly enough structure to compile into +two regex alternations and a set-membership test, with no interpreter and no +model in between. + +**Word boundaries are load-bearing and are not re-implemented here.** Every +pattern comes from :func:`src.matching.entity_matcher.compile_terms`, which owns +``LEFT_BOUNDARY`` / ``RIGHT_BOUNDARY`` and the shouted-acronym case rule. The +numbers that justify the anchoring were measured on this repository's own +corpus: across 55,249 articles ``nist`` occurs as a substring in 5,364 titles +and at a word boundary in 73; ``mail`` is 1,842 against 339. A second matcher in +this file would be a second place for that factor-of-seventy to come back. + +**Entities have no alias registry yet, and this file does not invent one.** The +beat-file loader that used to declare a :class:`CoverageEntity`'s aliases died +with the briefing product (task 012), and nothing has replaced it. So an entity +named in a trigger clause is matched by the surface form the operator wrote and +by nothing else. ``terms`` and ``entities`` therefore match by identical rules +today; they remain separate fields because they are separate *conjuncts* -- +"FedRAMP PMO AND (continuous monitoring OR ConMon)" is not expressible in one +list -- and because an alias registry plugs in at exactly one place, +:func:`_entity_terms`, when one exists. + +Added 2026-08-31 for backlog task 015. +""" + +from __future__ import annotations + +import re +from collections.abc import Iterable, Mapping, Sequence +from dataclasses import dataclass +from typing import Any + +from ..matching.entity_matcher import compile_terms, matches_any + +# Imported from the validator rather than restated, so that the compiler and the +# loader cannot disagree about which three axes a clause may constrain. +from ..position._validate import TRIGGER_FIELDS + +__all__ = [ + "TRIGGER_FIELDS", + "CompiledClause", + "CompiledWatch", + "TriggerCompileError", + "compile_watch", + "compile_watches", + "source_keys", +] + + +class TriggerCompileError(ValueError): + """ + Raised when stored triggers cannot be compiled into a predicate. + + The YAML loader already rejects unusable triggers, but it is not the only + way a row reaches ``watches.triggers`` -- a hand-edited database, a restored + dump, or a future migration can all put JSON there. A clause that constrains + nothing matches *every* observation, which is the single most expensive + mistake this tier can make, so it is refused loudly here rather than + evaluated. + """ + + +def _normalize_source(value: str) -> str: + """The comparison form of a source name or URL: trimmed and case-folded.""" + return value.strip().casefold() + + +def source_keys(*values: str | None) -> frozenset[str]: + """ + Every name an observation's source answers to, in comparison form. + + A ``sources`` clause is an *allowlist*, matched by equality rather than by + substring: "Federal Register" must not be satisfied by a feed called + "Federal Register Watch Blog". Callers pass whatever identifies the source + -- the feed's name, the feed's URL, the ``source_url`` recorded in the + payload -- and any one of them matching is a match. + """ + return frozenset(_normalize_source(v) for v in values if v and v.strip()) + + +def _entity_terms(entity: str) -> tuple[str, ...]: + """ + The surface forms of one entity named in a trigger clause. + + Today: the name as written, and nothing else. See the module docstring -- + there is no alias registry in the repository, and inventing surface forms + the operator did not write is how a routing predicate widens without anyone + editing a trigger. When an alias registry lands, it lands here. + """ + return (entity,) + + +@dataclass(frozen=True) +class CompiledClause: + """ + One conjunctive clause, compiled. Evaluated with regexes and a set test. + + ``index`` is the clause's position in the watch's ``triggers`` list, so a + routing decision can name the line of the file that produced it. + """ + + index: int + terms: tuple[re.Pattern[str], ...] + entities: tuple[re.Pattern[str], ...] + sources: frozenset[str] + declared: Mapping[str, tuple[str, ...]] + + def matches(self, text: str, observation_sources: frozenset[str]) -> bool: + """ + Whether this clause fires. Every populated field must match. + + Order is cheapest-first: the source test is a set intersection and + rejects a whole feed's worth of observations without touching a regex. + """ + if self.sources and not (self.sources & observation_sources): + return False + if self.terms and not matches_any(self.terms, text): + return False + return not self.entities or matches_any(self.entities, text) + + def describe(self) -> str: + """The clause as it was declared, for a dry-run listing.""" + parts = [ + f"{name}=[{', '.join(values)}]" for name, values in self.declared.items() if values + ] + return " AND ".join(parts) + + +@dataclass(frozen=True) +class CompiledWatch: + """ + One watch's triggers, compiled. ``matches`` returns the clause that fired. + + Returning the clause index rather than a bool is what makes a too-loose + trigger attributable: "watch X routed 400 observations" is a bill, "clause 2 + of watch X routed 400 observations" is a diff. + """ + + watch_id: str + clauses: tuple[CompiledClause, ...] + + def matching_clause(self, text: str, observation_sources: frozenset[str]) -> int | None: + """The index of the first clause that fires, or None. First in file order.""" + for clause in self.clauses: + if clause.matches(text, observation_sources): + return clause.index + return None + + def matches(self, text: str, observation_sources: frozenset[str]) -> bool: + return self.matching_clause(text, observation_sources) is not None + + +def _clause_values(raw: Any, field: str, label: str) -> tuple[str, ...]: + """One field of a stored clause, as a tuple of non-blank strings.""" + value = raw.get(field) + if value is None: + return () + if isinstance(value, str) or not isinstance(value, Sequence): + raise TriggerCompileError( + f"{label}: '{field}' must be a list of strings, got {type(value).__name__}. " + f"A trigger is compiled into a predicate, so it cannot be a sentence." + ) + values = tuple(str(item).strip() for item in value if str(item).strip()) + if value and not values: + # The field was written down and every value in it was blank. Dropping + # it would remove a conjunct: `{entities: [" "], terms: [FedRAMP]}` + # reads as "FedRAMP AND some entity" and would evaluate as "FedRAMP", + # which is a broader watch than the one on the page. + raise TriggerCompileError( + f"{label}: '{field}' is {list(value)} -- every entry is blank, so the constraint " + f"would be dropped and the clause would be broader than it reads." + ) + return values + + +def compile_watch(watch_id: str, triggers: Any) -> CompiledWatch: + """ + Compile one watch's stored ``triggers`` JSON into a predicate. + + Raises :class:`TriggerCompileError` rather than skipping anything it cannot + compile. A silently skipped clause is a watch that has stopped watching, and + the only symptom is a staleness alert weeks later; a silently *kept* empty + clause is a watch that matches everything. Neither is allowed to happen + quietly. + """ + if not isinstance(triggers, list) or not triggers: + raise TriggerCompileError( + f"watch '{watch_id}': triggers must be a non-empty list of clauses, got " + f"{type(triggers).__name__}. A watch nothing can trigger never fires." + ) + + clauses: list[CompiledClause] = [] + for index, raw in enumerate(triggers): + label = f"watch '{watch_id}' triggers[{index}]" + if not isinstance(raw, dict): + raise TriggerCompileError(f"{label}: expected a mapping, got {type(raw).__name__}") + + unknown = sorted(set(raw) - set(TRIGGER_FIELDS)) + if unknown: + raise TriggerCompileError( + f"{label}: unknown field(s) {unknown}; a clause may constrain only " + f"{list(TRIGGER_FIELDS)}" + ) + + declared = {field: _clause_values(raw, field, label) for field in TRIGGER_FIELDS} + if not any(declared.values()): + raise TriggerCompileError( + f"{label}: constrains nothing. A clause with no terms, entities or sources " + f"matches every observation, which is the most expensive possible bug in " + f"this tier." + ) + + terms = compile_terms(declared["terms"]) + entity_forms = [form for name in declared["entities"] for form in _entity_terms(name)] + entities = compile_terms(entity_forms) + + # compile_terms drops blank surface forms. If a field was declared and + # every one of its values vanished, the clause silently lost a conjunct + # and is now broader than it reads. + for name, compiled in (("terms", terms), ("entities", entities)): + if declared[name] and not compiled: + raise TriggerCompileError( + f"{label}: '{name}' was declared as {list(declared[name])} but compiled to " + f"no pattern, which would drop the constraint and widen the clause." + ) + + clauses.append( + CompiledClause( + index=index, + terms=terms, + entities=entities, + sources=source_keys(*declared["sources"]), + declared=declared, + ) + ) + + return CompiledWatch(watch_id=watch_id, clauses=tuple(clauses)) + + +def compile_watches(rows: Iterable[Any]) -> list[CompiledWatch]: + """ + Compile every watch row, in id order. One bad row fails the whole call. + + Refusing the batch is deliberate: routing half the watch set would produce a + report that looks complete, and the watches missing from it are exactly the + ones whose triggers are broken. + """ + return [compile_watch(row.id, row.triggers) for row in sorted(rows, key=lambda r: r.id)] diff --git a/src/routing/router.py b/src/routing/router.py new file mode 100644 index 0000000..7f7e535 --- /dev/null +++ b/src/routing/router.py @@ -0,0 +1,202 @@ +""" +Running the compiled predicates over stored observations, and recording what matched. + +Two functions and a hard line between them, on the same pattern as replay: +:func:`route` computes and writes nothing, :func:`persist` writes. The dry run +is not a special mode of the write path -- it *is* the computation, and the +write path is a separate call that takes its result. A reporting mode that +shares a code path with the writer is a reporting mode that eventually writes. + +**Reading order is fixed.** Observations are considered newest-first by +``observed_at`` with ``content_hash`` breaking ties, so "the last N +observations" is a deterministic set and not a query plan. Watches are compiled +in id order. + +**Expired watches are not routed.** The architectural claim is that volume +scales with *live* watches; a watch whose ``expires`` has passed is not live, +and routing to it would spend the adjudicator's budget on a question that +stopped mattering. The count of skipped watches is reported rather than +swallowed, because "nothing routed today" and "every watch expired last week" +must not look the same. + +**This tier reads ``observations`` and never ``articles``.** The 55,249 +pre-rewrite rows are the legacy archive; the rule is written in +``src/database/models.py``. There is no import of ``Article`` in this package, +and tests/routing/test_no_model.py asserts it. + +Added 2026-08-31 for backlog task 015. +""" + +from __future__ import annotations + +import logging +from collections.abc import Sequence +from dataclasses import dataclass, field +from datetime import date +from typing import Any + +from sqlalchemy.orm import Session + +from ..database.models import Observation, RouteCandidate, RSSFeed, Watch +from ..sources.observation import payload_text +from .predicate import CompiledWatch, compile_watches, source_keys + +logger = logging.getLogger(__name__) + +__all__ = [ + "RoutedLink", + "RoutingReport", + "persist", + "route", +] + + +@dataclass(frozen=True, order=True) +class RoutedLink: + """One observation selected as candidate evidence for one watch.""" + + observation_hash: str + watch_id: str + clause_index: int + + +@dataclass +class RoutingReport: + """ + What routing would do. Produced by :func:`route`, which writes nothing. + + ``unrouted`` carries hashes rather than a count, because the count alone is + useless: it says a gap exists and nothing about where. Clustering them is + :mod:`src.routing.gaps`. + """ + + considered: int = 0 + watch_ids: tuple[str, ...] = () + expired_watch_ids: tuple[str, ...] = () + links: list[RoutedLink] = field(default_factory=list) + unrouted: list[str] = field(default_factory=list) + per_watch: dict[str, int] = field(default_factory=dict) + per_clause: dict[tuple[str, int], int] = field(default_factory=dict) + + @property + def routed_count(self) -> int: + """Observations with at least one link. Not the number of links.""" + return len({link.observation_hash for link in self.links}) + + @property + def unrouted_count(self) -> int: + return len(self.unrouted) + + @property + def fan_out(self) -> float: + """ + Links per observation considered -- what the adjudicator actually pays for. + + One observation routed to three watches is three adjudications. This is + the number the ceiling test pins, alongside the routed-observation + count, because they diverge exactly when a term is added to several + watches at once. + """ + return len(self.links) / self.considered if self.considered else 0.0 + + +def _source_lookup(db: Session) -> dict[int, tuple[str | None, str | None]]: + """``{feed id: (name, url)}`` for every feed, read once rather than per observation.""" + return {row.id: (row.name, row.url) for row in db.query(RSSFeed.id, RSSFeed.name, RSSFeed.url)} + + +def route( + db: Session, + *, + limit: int | None = None, + today: date | None = None, + watches: Sequence[Any] | None = None, +) -> RoutingReport: + """ + Offer the last ``limit`` observations to every live watch. Writes nothing. + + ``watches`` overrides the stored watch set, which is what the ceiling test + uses to hold the watch side fixed. ``today`` decides which watches count as + live; it defaults to the real date, and tests pass their own so the suite + does not expire. + """ + stamp = today or date.today() + rows = list(watches) if watches is not None else db.query(Watch).all() + + live: list[Any] = [] + expired: list[Any] = [] + for row in rows: + (live if row.expires is None or row.expires >= stamp else expired).append(row) + compiled: list[CompiledWatch] = compile_watches(live) + + report = RoutingReport( + watch_ids=tuple(watch.watch_id for watch in compiled), + expired_watch_ids=tuple(sorted(row.id for row in expired)), + per_watch={watch.watch_id: 0 for watch in compiled}, + ) + + feeds = _source_lookup(db) + + query = db.query(Observation.content_hash, Observation.source_id, Observation.payload).order_by( + Observation.observed_at.desc(), Observation.content_hash + ) + if limit is not None: + query = query.limit(limit) + + for content_hash, source_id, payload in query: + report.considered += 1 + text = payload_text(payload or {}) + name, url = feeds.get(source_id, (None, None)) + sources = source_keys(name, url, (payload or {}).get("source_url")) + + matched = False + for watch in compiled: + index = watch.matching_clause(text, sources) + if index is None: + continue + matched = True + report.links.append(RoutedLink(content_hash, watch.watch_id, index)) + report.per_watch[watch.watch_id] += 1 + key = (watch.watch_id, index) + report.per_clause[key] = report.per_clause.get(key, 0) + 1 + if not matched: + report.unrouted.append(content_hash) + + report.links.sort() + return report + + +def persist(db: Session, report: RoutingReport) -> dict[str, int]: + """ + Write ``report``'s links as :class:`RouteCandidate` rows. Idempotent. + + Returns ``{"inserted": n, "already_linked": m}``. Re-running routing over an + unchanged corpus inserts nothing: Tier 1 is deterministic, so the second run + produces the same links, and every one of them is already there. + + The pre-read is what makes the *report* honest -- it is how the caller is + told nothing was inserted. The database's unique constraint is what makes + the *table* honest, and it is not removable: two concurrent routing runs + would both pass this check. + """ + existing = { + (row.observation_hash, row.watch_id) + for row in db.query(RouteCandidate.observation_hash, RouteCandidate.watch_id) + } + + inserted = 0 + for link in report.links: + if (link.observation_hash, link.watch_id) in existing: + continue + db.add( + RouteCandidate( + observation_hash=link.observation_hash, + watch_id=link.watch_id, + clause_index=link.clause_index, + ) + ) + existing.add((link.observation_hash, link.watch_id)) + inserted += 1 + + db.flush() + return {"inserted": inserted, "already_linked": len(report.links) - inserted} diff --git a/src/routing/salience.py b/src/routing/salience.py new file mode 100644 index 0000000..ed25893 --- /dev/null +++ b/src/routing/salience.py @@ -0,0 +1,183 @@ +""" +Labelling a coverage gap: which words the unrouted observations are about. + +Split out of :mod:`src.routing.gaps` on 2026-08-31 (backlog task 015) to keep +both files inside the repository's 200-300 line rule. The seam is real: +everything here is a pure function of text and knows nothing about +observations, watches, clusters or the database. + +**Why this exists at all.** Near-duplicate clustering catches one story carried +by six outlets. It cannot catch a subject covered by forty *different* stories +over a month, no two of which are near-duplicates -- and that is the shape a +missing sensor usually has. Document frequency over salient words catches it. +Frequency is counted once per document, so one verbose article cannot invent a +theme. + +This is a label, not a topic model. It is deliberately the crudest thing that +makes the unrouted set readable, because the decision it feeds -- "is there a +watch missing here?" -- is a human one (backlog task 021), and a cleverer +grouping would mostly be harder to argue with. +""" + +from __future__ import annotations + +import re +from collections import Counter + +__all__ = ["MAX_GAP_TERMS", "gap_terms"] + +_WORD = re.compile(r"[0-9A-Za-z][0-9A-Za-z'-]*") + +# Words that carry no subject. Deliberately short: the length floor below +# already removes most function words, and a long hand-tuned stoplist is a place +# to accidentally suppress a real subject. Anything not here is reported, and +# whoever reads the file can see the noise for what it is. +_STOPWORDS = frozenset( + [ + "about", + "after", + "again", + "against", + "also", + "among", + "another", + "because", + "been", + "before", + "being", + "between", + "both", + "cannot", + "could", + "does", + "doing", + "done", + "down", + "during", + "each", + "either", + "else", + "even", + "ever", + "every", + "from", + "further", + "had", + "has", + "have", + "having", + "here", + "how", + "however", + "into", + "itself", + "just", + "like", + "made", + "make", + "many", + "may", + "might", + "more", + "most", + "much", + "must", + "never", + "new", + "news", + "now", + "only", + "other", + "others", + "our", + "out", + "over", + "own", + "report", + "reports", + "said", + "same", + "say", + "says", + "see", + "seen", + "should", + "since", + "some", + "such", + "than", + "that", + "their", + "them", + "then", + "there", + "these", + "they", + "this", + "those", + "through", + "time", + "today", + "too", + "under", + "until", + "upon", + "used", + "using", + "very", + "want", + "was", + "way", + "well", + "were", + "what", + "when", + "where", + "which", + "while", + "who", + "whom", + "why", + "will", + "with", + "within", + "without", + "would", + "year", + "years", + "your", + ] +) + +# Four, because "NIST", "CISA", "OMB"... -- three-letter acronyms are the +# shortest thing that can name a subject, and a three-letter floor drags in +# every remaining function word. The stoplist absorbs the four-letter ones that +# survive. A subject named only by a three-letter acronym will show up in the +# near-duplicate clusters instead; it is not lost, only unlabelled. +_MIN_TERM_LENGTH = 4 + +# How many gap terms to write. The tail is a long list of hapaxes; it is not +# information, it is the corpus. +MAX_GAP_TERMS = 50 + + +def _salient(text: str) -> set[str]: + """The subject-bearing words of one text, deduplicated within the document.""" + return { + word + for token in _WORD.findall(text.lower()) + if len(word := token.strip("'-")) >= _MIN_TERM_LENGTH and word not in _STOPWORDS + } + + +def gap_terms(texts: list[str], limit: int = MAX_GAP_TERMS) -> list[tuple[str, int]]: + """ + Salient words by document frequency, most common first. + + Ties break alphabetically so the report is a deterministic function of the + corpus rather than of dict insertion order. + """ + counts: Counter[str] = Counter() + for text in texts: + counts.update(_salient(text)) + return sorted(counts.items(), key=lambda item: (-item[1], item[0]))[:limit] diff --git a/src/sources/observation.py b/src/sources/observation.py index 0cd3c65..02dd630 100644 --- a/src/sources/observation.py +++ b/src/sources/observation.py @@ -58,6 +58,7 @@ "observation_hash", "observation_payload", "observe", + "payload_text", "store_observation", ] @@ -112,18 +113,31 @@ def observation_hash(payload: dict[str, Any]) -> str: return content_hash(*(_render(payload[key]) for key in OBSERVATION_FIELDS)) -def minhash_text(payload: dict[str, Any]) -> str: +def payload_text(payload: dict[str, Any]) -> str: """ - The text a near-duplicate signature is computed over: title plus body. + The readable text of a stored observation: title plus body. ``normalized_content`` is preferred over ``description`` because it is the HTML-stripped full text; ``description`` is the fallback for feeds that - carry only a summary. + carry only a summary. Raw ``content`` is never used -- it still carries + markup, and a word-boundary matcher pointed at markup matches attribute + values. + + Every deterministic reader of an observation reads *this*. Near-duplicate + signatures did first; Tier 1 routing joined on 2026-08-31 (backlog task + 015). Two definitions of "the text of an observation" would let a + near-duplicate group and a routing decision disagree about what a document + says. """ body = payload.get("normalized_content") or payload.get("description") or "" return f"{payload.get('title') or ''} {body}".strip() +def minhash_text(payload: dict[str, Any]) -> str: + """The text a near-duplicate signature is computed over. See :func:`payload_text`.""" + return payload_text(payload) + + @dataclass(frozen=True) class ObservationRecord: """A hashed, signed observation, not yet stored.""" diff --git a/src/sources/rss_adapter.py b/src/sources/rss_adapter.py index b7628b0..e9416f6 100644 --- a/src/sources/rss_adapter.py +++ b/src/sources/rss_adapter.py @@ -2,13 +2,17 @@ The existing RSS fetcher, wrapped in the adapter contract. This module adds *no* behaviour: ``src/rss/fetcher.py`` is not modified and not -subclassed, and the live pipeline still fetches feeds through -``fetch_all_active_feeds`` exactly as before. What is new is a second way to -*read* an RSS feed -- returning :class:`~src.sources.base.RawItem` values -instead of writing rows -- so that RSS and the Federal Register API are the -same kind of thing to any caller. - -Added 2026-08-26 for backlog task 005. +subclassed. It offers a way to *read* an RSS feed -- returning +:class:`~src.sources.base.RawItem` values instead of writing rows -- so that +RSS and the Federal Register API are the same kind of thing to any caller. + +Added 2026-08-26 for backlog task 005. Corrected 2026-08-31: this used to say +the live pipeline still fetched through ``fetch_all_active_feeds`` "exactly as +before". That is no longer true and has not been since task 025 -- both it and +``RSSFetcher.fetch_and_store_feed`` now raise, because they wrote articles with +no observation and Tier 1 routing cannot see those. **This adapter, through +``src.sources.store.store_items``, is the only way an RSS article enters the +corpus.** """ from __future__ import annotations diff --git a/tests/cli/test_route_cli.py b/tests/cli/test_route_cli.py new file mode 100644 index 0000000..c775502 --- /dev/null +++ b/tests/cli/test_route_cli.py @@ -0,0 +1,230 @@ +""" +Tests for the ``route`` CLI command (backlog task 015). + +The acceptance criterion is specific about what ``--dry-run`` has to report: +per watch, how many of the last N observations would route, **and** the unrouted +count **and its clusters**. The last part is the one that is easy to drop, and +it is the whole coverage-gap signal -- so it is asserted on the rendered output +rather than on the report object the command happens to build. +""" + +from contextlib import contextmanager +from datetime import date, datetime +from pathlib import Path +from unittest.mock import patch + +import pytest +from click.testing import CliRunner + +from src.cli.route import route_command +from src.database.models import RouteCandidate, RSSFeed +from src.database.models import Watch as WatchRow +from src.sources.base import RawItem +from src.sources.observation import store_observation + +WIRE = "The county board voted on Tuesday to approve the stormwater levy after a hearing." + + +def _patch_db(session): + @contextmanager + def _ctx(): + try: + yield session + session.commit() + except Exception: + session.rollback() + raise + + return patch("src.cli.route.get_db", _ctx) + + +@pytest.fixture +def corpus(test_session): + """Two feeds, four observations, one live watch and one expired one.""" + scoop = RSSFeed(url="https://fedscoop.test/rss", name="FedScoop") + cardinal = RSSFeed(url="https://cardinal.test/rss", name="Cardinal News") + test_session.add_all([scoop, cardinal]) + test_session.flush() + + items = [ + (scoop, "CISA issues an advisory", "CISA published an advisory today."), + (scoop, "County approves stormwater levy", WIRE), + (cardinal, "County approves stormwater levy", WIRE + " Reporting by staff."), + (cardinal, "Ferry service resumes", "The harbour authority restored the timetable."), + ] + for index, (feed, title, body) in enumerate(items): + store_observation( + test_session, + feed, + RawItem( + guid=f"g{index}", + url=f"https://example.test/{index}", + title=title, + normalized_content=body, + published_date=datetime(2026, 8, 20), + ), + ) + + test_session.add_all( + [ + WatchRow( + id="cisa-advisories", + claim="CISA keeps issuing advisories.", + belief=0.6, + decision_key="fedramp-authorization-renewal", + so_what="It sets the remediation workload.", + triggers=[{"entities": ["CISA"], "terms": ["advisory"]}], + expires=date(2027, 3, 31), + staleness_alert_days=30, + ), + WatchRow( + id="expired-question", + claim="Something that stopped mattering.", + belief=0.5, + decision_key="fedramp-authorization-renewal", + so_what="It no longer bears on anything.", + triggers=[{"terms": ["ferry"]}], + expires=date(2026, 1, 1), + staleness_alert_days=30, + ), + ] + ) + test_session.flush() + return test_session + + +def _run(session, args, tmp_path): + with _patch_db(session): + return CliRunner().invoke(route_command, [*args, "--gaps-out", str(tmp_path / "gaps.json")]) + + +class TestDryRunReporting: + def test_it_reports_per_watch_how_many_would_route(self, corpus, tmp_path): + result = _run(corpus, ["--dry-run"], tmp_path) + + assert result.exit_code == 0, result.output + assert "cisa-advisories" in result.output + assert "4 of 4 stored observation(s)" in result.output + + def test_it_reports_the_unrouted_count(self, corpus, tmp_path): + result = _run(corpus, ["--dry-run"], tmp_path) + + assert "unrouted" in result.output + assert "routed" in result.output + assert "3" in result.output + + def test_it_reports_the_unrouted_clusters_not_just_the_count(self, corpus, tmp_path): + """ + The landmine. A bare integer makes the coverage-gap signal useless. + + The two copies of the county wire story are near-duplicates and must be + shown as one cluster of two, with a subject attached. + """ + result = _run(corpus, ["--dry-run"], tmp_path) + + assert "UNROUTED CLUSTERS" in result.output + assert "County approves stormwater levy" in result.output + assert "stormwater" in result.output + assert "gap terms:" in result.output + + def test_it_names_the_clause_that_fired(self, corpus, tmp_path): + result = _run(corpus, ["--dry-run"], tmp_path) + assert "clause 0:" in result.output + + def test_it_names_the_watches_it_skipped_as_expired(self, corpus, tmp_path): + result = _run(corpus, ["--dry-run"], tmp_path) + assert "expired-question" in result.output + assert "skipped 1 expired" in result.output + + def test_it_writes_the_gap_report_where_task_021_can_read_it(self, corpus, tmp_path): + import json + + result = _run(corpus, ["--dry-run"], tmp_path) + target = tmp_path / "gaps.json" + + assert target.exists() + assert str(target) in result.output + payload = json.loads(target.read_text()) + assert payload["unrouted_observations"] == 3 + assert [c["size"] for c in payload["clusters"]] == [2, 1] + + def test_dry_run_writes_no_route_candidates(self, corpus, tmp_path): + _run(corpus, ["--dry-run"], tmp_path) + assert corpus.query(RouteCandidate).count() == 0 + + def test_limit_bounds_what_is_considered(self, corpus, tmp_path): + result = _run(corpus, ["--dry-run", "--limit", "2"], tmp_path) + assert "2 of 4 stored observation(s)" in result.output + + +class TestWriting: + def test_without_dry_run_it_records_the_links(self, corpus, tmp_path): + result = _run(corpus, [], tmp_path) + + assert result.exit_code == 0, result.output + assert "1 link(s) inserted, 0 already present" in result.output + assert corpus.query(RouteCandidate).count() == 1 + + def test_running_it_twice_records_them_once(self, corpus, tmp_path): + _run(corpus, [], tmp_path) + result = _run(corpus, [], tmp_path) + + assert "0 link(s) inserted, 1 already present" in result.output + assert corpus.query(RouteCandidate).count() == 1 + + +class TestRefusals: + def test_it_says_so_when_there_are_no_observations(self, test_session, tmp_path): + result = _run(test_session, ["--dry-run"], tmp_path) + assert result.exit_code == 0 + assert "No observations are stored" in result.output + assert "never 'articles'" in result.output + + def test_it_says_so_when_there_are_no_watches(self, test_session, tmp_path): + feed = RSSFeed(url="https://fedscoop.test/rss", name="FedScoop") + test_session.add(feed) + test_session.flush() + store_observation( + test_session, + feed, + RawItem(guid="g", url="u", title="anything", normalized_content="text"), + ) + + result = _run(test_session, ["--dry-run"], tmp_path) + assert "No watches are stored" in result.output + + def test_an_uncompilable_trigger_stops_the_command(self, corpus, tmp_path): + corpus.query(WatchRow).filter_by(id="cisa-advisories").one().triggers = [{}] + corpus.flush() + + result = _run(corpus, ["--dry-run"], tmp_path) + assert result.exit_code != 0 + assert "constrains nothing" in str(result.output) + str(result.exception) + + +class TestNoModelFlagExists: + """ + Tier 1 takes no model, no prompt version and no adjudicator. + + Checked against the command's registered parameters rather than against + anyone's intention, on the same pattern as ``watch``'s invariant-6 test. + """ + + def test_the_command_has_no_model_shaped_option(self): + names = {param.name for param in route_command.params} + assert names == {"dry_run", "limit", "gaps_out", "no_gaps"} + + +def test_route_is_registered_on_the_cli(): + from src.cli.app import COMMAND_DISPATCH, cli + + assert COMMAND_DISPATCH["route"] is route_command + assert "route" in cli.commands + + +def test_the_default_gap_path_is_inside_the_data_directory(): + from src.routing.gaps import default_gap_report_path + + path = default_gap_report_path() + assert path.parts[-2:] == ("routing", "unrouted_clusters.json") + assert Path(path).is_absolute() diff --git a/tests/database/test_add_route_candidates.py b/tests/database/test_add_route_candidates.py new file mode 100644 index 0000000..bb93c62 --- /dev/null +++ b/tests/database/test_add_route_candidates.py @@ -0,0 +1,80 @@ +""" +Tests for the migration that adds ``route_candidates`` (backlog task 015). + +Weighted, like the observations migration's tests, towards the property that +would be silently lost if the model-driven creation were replaced by hand-written +DDL. Here that property is the unique constraint on +``(observation_hash, watch_id)``: a table created without it looks correct in +every way except that routing stops being idempotent, and the symptom is a +doubled candidate count on the second run rather than an error. +""" + +import pytest +from sqlalchemy import create_engine, inspect, text + +from src.database.migrations import add_route_candidates as mig +from src.database.models import Base, RouteCandidate + + +@pytest.fixture +def blank_engine(tmp_path): + return create_engine(f"sqlite:///{tmp_path / 'fresh.db'}") + + +@pytest.fixture +def corpus_engine(tmp_path): + """A database that already has the rest of the schema, as the real one does.""" + engine = create_engine(f"sqlite:///{tmp_path / 'corpus.db'}") + Base.metadata.create_all(bind=engine) + with engine.begin() as conn: + conn.execute(text("DROP TABLE route_candidates")) + return engine + + +class TestUpgrade: + def test_it_creates_the_table(self, blank_engine): + assert mig.upgrade(blank_engine) == ["route_candidates"] + assert inspect(blank_engine).has_table("route_candidates") + + def test_it_is_a_no_op_when_the_table_exists(self, corpus_engine): + mig.upgrade(corpus_engine) + assert mig.upgrade(corpus_engine) == [] + + def test_it_does_not_touch_articles_or_observations(self, corpus_engine): + before = set(inspect(corpus_engine).get_table_names()) + mig.upgrade(corpus_engine) + after = set(inspect(corpus_engine).get_table_names()) + assert after - before == {"route_candidates"} + + +class TestTheConstraintThatMakesRoutingIdempotent: + def test_the_unique_constraint_is_present(self, blank_engine): + mig.upgrade(blank_engine) + constraints = inspect(blank_engine).get_unique_constraints("route_candidates") + assert any( + set(c["column_names"]) == {"observation_hash", "watch_id"} for c in constraints + ), constraints + + def test_the_clause_index_check_is_present(self, blank_engine): + mig.upgrade(blank_engine) + names = {c["name"] for c in inspect(blank_engine).get_check_constraints("route_candidates")} + assert "ck_route_clause_index" in names + + def test_the_model_and_the_migration_cannot_drift(self, blank_engine): + """The migration creates from the model, so this is a tautology by design.""" + mig.upgrade(blank_engine) + created = {c["name"] for c in inspect(blank_engine).get_columns("route_candidates")} + assert created == {c.name for c in RouteCandidate.__table__.columns} + + +class TestDowngrade: + def test_it_refuses_without_confirmation(self, blank_engine): + mig.upgrade(blank_engine) + with pytest.raises(SystemExit): + mig.downgrade(blank_engine) + assert inspect(blank_engine).has_table("route_candidates") + + def test_it_drops_the_table_when_confirmed(self, blank_engine): + mig.upgrade(blank_engine) + assert mig.downgrade(blank_engine, confirmed=True) == ["route_candidates"] + assert not inspect(blank_engine).has_table("route_candidates") diff --git a/tests/database/test_connection.py b/tests/database/test_connection.py index 2af0b34..df19c04 100644 --- a/tests/database/test_connection.py +++ b/tests/database/test_connection.py @@ -120,13 +120,15 @@ def test_create_tables_creates_all_tables(self, tmp_path, mocker): # nineteen models, and a subset check would pass again the moment one of # them was reintroduced by accident. `watches` was added deliberately by # task 013 (2026-08-31); `observations` and `evidence` by task 014 the - # same day. Every addition here should be a task that says so. + # same day; `route_candidates` by task 015, also 2026-08-31. Every + # addition here should be a task that says so. assert set(table_names) == { "rss_feeds", "articles", "watches", "observations", "evidence", + "route_candidates", } def test_create_tables_is_idempotent(self, tmp_path, mocker): diff --git a/tests/database/test_models.py b/tests/database/test_models.py index ee3420e..45c177d 100644 --- a/tests/database/test_models.py +++ b/tests/database/test_models.py @@ -191,7 +191,8 @@ def test_only_the_declared_tables_are_mapped(self): mapped set is pinned rather than left to review. ``watches`` joined the set on 2026-08-31 (backlog task 013); - ``observations`` and ``evidence`` on 2026-08-31 (backlog task 014). + ``observations`` and ``evidence`` on 2026-08-31 (backlog task 014); + ``route_candidates`` on 2026-08-31 (backlog task 015, Tier 1 routing). Every addition to this set should be a task that says so. """ from src.database.models import Base @@ -202,4 +203,5 @@ def test_only_the_declared_tables_are_mapped(self): "watches", "observations", "evidence", + "route_candidates", } diff --git a/tests/routing/__init__.py b/tests/routing/__init__.py new file mode 100644 index 0000000..e69de29 diff --git a/tests/routing/conftest.py b/tests/routing/conftest.py new file mode 100644 index 0000000..e90b16b --- /dev/null +++ b/tests/routing/conftest.py @@ -0,0 +1,85 @@ +""" +Shared fixtures for the Tier 1 routing tests. + +Observations here are built through the real write path +(``src.sources.observation.store_observation``), not by inserting rows. A test +corpus assembled by hand is a corpus whose hashes, minhash signatures and +payload shape are whatever the test author assumed, and the routing code reads +all three. +""" + +from __future__ import annotations + +from datetime import date, datetime, timedelta + +import pytest + +from src.database.models import RSSFeed +from src.sources.base import RawItem +from src.sources.observation import store_observation + +TODAY = date(2026, 8, 31) + + +class WatchRow: + """ + The three attributes :func:`src.routing.router.route` reads off a watch. + + Deliberately not a :class:`src.database.models.Watch`: the router must not + quietly grow a dependency on a column it was not given, and this fails loudly + if it does. + """ + + def __init__(self, watch_id: str, triggers, expires: date = date(2027, 1, 1)): + self.id = watch_id + self.triggers = triggers + self.expires = expires + + +@pytest.fixture +def feed(test_session): + """One stored feed, so observations have a source to belong to.""" + + def _feed(name="FedScoop", url=None): + row = RSSFeed( + url=url or f"https://example.test/{name.lower().replace(' ', '-')}", name=name + ) + test_session.add(row) + test_session.flush() + return row + + return _feed + + +@pytest.fixture +def observe(test_session, feed): + """ + Store one observation with the given title and body. Returns its hash. + + ``observed_at`` is left to the insert default. It cannot be set afterwards: + observations are immutable, and both the ORM guard and a SQLite trigger + refuse the UPDATE. Tests here therefore assert on *how many* of the last N + were considered rather than on which -- ``utcnow()`` has microsecond + resolution, so insertion order and ``observed_at`` order agree, but that is + a property of the clock and not something a test should lean on. + """ + default_feed: list[RSSFeed] = [] + counter = [0] + + def _observe(title, body="", *, source=None, published=None): + if source is None: + if not default_feed: + default_feed.append(feed()) + source = default_feed[0] + counter[0] += 1 + item = RawItem( + guid=f"guid-{counter[0]}", + url=f"https://example.test/item/{counter[0]}", + title=title, + normalized_content=body, + published_date=published or datetime(2026, 8, 20) + timedelta(minutes=counter[0]), + ) + content_hash, _ = store_observation(test_session, source, item) + return content_hash + + return _observe diff --git a/tests/routing/fixtures/ceiling_corpus.jsonl.gz b/tests/routing/fixtures/ceiling_corpus.jsonl.gz new file mode 100644 index 0000000..927c5c8 Binary files /dev/null and b/tests/routing/fixtures/ceiling_corpus.jsonl.gz differ diff --git a/tests/routing/fixtures/ceiling_watches.yaml b/tests/routing/fixtures/ceiling_watches.yaml new file mode 100644 index 0000000..73246b6 --- /dev/null +++ b/tests/routing/fixtures/ceiling_watches.yaml @@ -0,0 +1,118 @@ +# The five watches the Tier 1 ceiling test routes against. +# +# NOT an operator watch set and not a proposal for one. It exists so that the +# ceiling in tests/routing/test_ceiling.py is measured against a *fixed* watch +# side: if the watch set could drift, a widening in the routing predicate and a +# widening in the triggers would be indistinguishable, and the test would stop +# being a gate on the thing it is meant to gate. +# +# Provenance, so the number the ceiling rests on can be argued with: +# +# * The first three watches are copied verbatim from +# config/watches.example.yaml -- the hand-authored, checked-in set. Their +# triggers were not touched (tuning the shipped triggers is out of scope for +# backlog task 015). +# * The last two were written for this fixture on 2026-08-31, in the same +# domain and the same shape, because the acceptance criterion names five +# watches and the shipped set has three. They are pointed at subjects the +# measurement corpus actually carries (CISA advisories, NIST publications) +# so that the measured baseline is not identically zero -- a ceiling over a +# baseline of zero gates only against catastrophic widening. The three +# shipped watches above route zero against that corpus, which is a real +# finding about the feed list and not a defect in this fixture. +# +# Decision keys are the three in config/position.example.yaml, because the +# loader refuses a watch whose so_what.decision names no decision, and the test +# loads this file through the real loader rather than around it. + +version: 1 + +watches: + - id: fedramp-conmon-scope-expands + claim: > + FedRAMP expands continuous-monitoring requirements for Moderate systems in + a way that adds recurring work, before the 2027 assessment window. + belief: 0.35 + so_what: + decision: fedramp-authorization-renewal + because: > + Scope expansion moves renewal from two engineer-quarters to three or + more, which is the point at which lapsing becomes the cheaper option. + triggers: + - entities: [FedRAMP PMO, GSA] + terms: [continuous monitoring, ConMon, significant change] + - sources: [Federal Register] + terms: [FedRAMP] + expires: 2027-03-31 + staleness_alert_days: 30 + + - id: fedramp-20x-supersedes-rev5 + claim: > + The FedRAMP 20x program replaces the Rev 5 baseline for new + authorizations, rather than running alongside it. + belief: 0.45 + so_what: + decision: fedramp-authorization-renewal + because: > + If 20x supersedes rather than supplements, the renewal is a + re-authorization against a different baseline and the estimate is wrong + by a factor, not a margin. + triggers: + - terms: [FedRAMP 20x, Rev 5, revision 5] + entities: [FedRAMP PMO] + expires: 2027-06-30 + staleness_alert_days: 3w + + - id: compliance-hiring-market-tightens + claim: > + Median time-to-hire for a compliance engineer in the DC market exceeds 90 + days through the end of 2026. + belief: 0.6 + so_what: + decision: hire-compliance-engineer + because: > + A 90-day search started in December lands after the assessment window + opens, which converts "hire later if needed" into "do not hire". + triggers: + - terms: [time to hire, compliance engineer, GRC hiring] + - entities: [OPM] + terms: [cyber workforce, hiring authority] + expires: 2026-12-15 + staleness_alert_days: 45 + + - id: cisa-kev-cadence-holds + claim: > + CISA keeps adding to the Known Exploited Vulnerabilities catalog at its + current cadence through 2027 rather than slowing it. + belief: 0.55 + so_what: + decision: fedramp-authorization-renewal + because: > + The KEV catalog is what the remediation-timeline control points at, so + its cadence is what the renewal's continuous-monitoring effort costs. + triggers: + - entities: [CISA] + terms: [Known Exploited Vulnerabilities, KEV, binding operational directive, + emergency directive] + - entities: [CISA] + terms: [advisory, advisories, alert] + expires: 2027-12-31 + staleness_alert_days: 21 + + - id: nist-800-53-revision-lands + claim: > + NIST issues a further revision to the SP 800-53 control catalog before the + 2027 assessment window. + belief: 0.4 + so_what: + decision: fedramp-authorization-renewal + because: > + A revised control catalog re-baselines the assessment, which is the + difference between a delta review and a full re-authorization. + triggers: + - entities: [NIST] + terms: [Special Publication, SP 800, 800-53, control catalog, revision] + - entities: [NIST, OMB] + terms: [zero trust architecture] + expires: 2027-09-30 + staleness_alert_days: 45 diff --git a/tests/routing/test_ceiling.py b/tests/routing/test_ceiling.py new file mode 100644 index 0000000..fba8c23 --- /dev/null +++ b/tests/routing/test_ceiling.py @@ -0,0 +1,279 @@ +""" +The ceiling: 1,000 observations against 5 watches route fewer than K. + +**Why this test exists.** "Most observations match nothing and die in Tier 1" is +an architectural claim, and an architectural claim that is only observable in +production metrics is one that regresses between releases and is noticed at the +invoice. A routing predicate that is too loose does not raise, does not log and +does not fail any test that only checks it matches what it should. This test +attaches a number to the claim and makes it a gate. + +================================================================================ +HOW K WAS MEASURED, AND AGAINST WHAT +================================================================================ + +**The corpus is real article text, not synthesised.** ``ceiling_corpus.jsonl.gz`` +holds 1,000 rows taken from the 55,249-article pre-rewrite archive in this +repository's own database, read through a read-only (``immutable=1``) SQLite +handle on 2026-08-31. Nothing was written back and nothing was backfilled; the +extraction was a measurement harness, and what survives it is this fixture plus +the numbers below. Each row carries the article's title, its body +(``normalized_content``, falling back to ``description``, truncated to 4,000 +characters), its publication date, and the name and URL of the feed that carried +it. The test loads them through the real write path, +``src.sources.observation.store_observation``, so the observations it routes have +real content hashes, real payload shape and real MinHash signatures. + +**The sample is stratified by feed category, deliberately not by trigger word.** +400 rows come from the feeds categorised as this operator's domain +(cybersecurity, security_news, security_alerts, security_analysis, +security_standards, federal_policy, federal_government, legislative, oversight, +budget, fiscal_policy, judicial -- 3,228 articles in total) and 600 from the +whole corpus. Stratifying on the trigger words instead would have been circular: +the corpus would have been defined by the answer. Stratifying on feed category +models what is actually true of an operator -- their feed list is weighted toward +their subject, so a day's ingestion is not a uniform sample of the news. +Selection within each stratum is by BLAKE2b of ``(article id, guid)``, so it is +deterministic and does not depend on row order or on a PRNG. + +**The watch side is fixed** by ``ceiling_watches.yaml``: three watches copied +verbatim from the shipped ``config/watches.example.yaml`` and two written for the +fixture. See that file for the provenance of each. + +**The measured baseline, 2026-08-31:** 5 of 1,000 observations route, producing +5 links, a fan-out of 0.0050 adjudications per observation considered. All 5 are +attributable to ``cisa-kev-cadence-holds`` (4 on clause 0, 1 on clause 1). The +three shipped watches route zero. That is a real finding about the feed list +rather than a defect in the fixture: across the *entire* 55,249-article corpus, +"fedramp" occurs as a substring in 5 articles and "cmmc" in 6, so a conjunctive +FedRAMP trigger has nothing to match. It is exactly the kind of thing the +unrouted-cluster report exists to make visible. + +**K = 20, and here is why that number and not another.** The four widening +regressions below were each measured against this same fixture on 2026-08-31 by +monkeypatching the shipped code: + + baseline (as shipped) 5 routed, 5 links + word boundaries deleted 5 routed, 5 links + shouted-acronym case rule deleted 5 routed, 5 links + boundaries and case both deleted 6 routed, 6 links + clause AND became OR 33 routed, 37 links + term/entity constraints dropped 1,000 routed, 5,000 links + +K = 20 sits at four times the baseline and below the cheapest of the two +regressions this test can see. It is a gate on *over-routing*, which is the +failure that costs money. + +**What this test does NOT gate, stated plainly.** Deleting the word boundaries +moves this number by zero, and deleting boundaries and case together moves it by +one. It cannot gate them, because every surface form in a realistic compliance +watch set is either a shouted acronym (protected by the case rule) or a phrase +with no common host word. Word-boundary regression is gated by +``tests/routing/test_predicate.py::TestWordBoundariesAreLoadBearing``, whose +cases are all non-shouted terms and which was verified to fail with the anchors +deleted. Saying this test covers boundaries would be false. + +Added 2026-08-31 for backlog task 015. +""" + +from __future__ import annotations + +import gzip +import json +from datetime import date, datetime +from pathlib import Path + +import pytest +from sqlalchemy import create_engine +from sqlalchemy.orm import sessionmaker + +from src.database.models import Base, RSSFeed +from src.position import load_position, load_watches +from src.routing import route +from src.sources.base import RawItem +from src.sources.observation import store_observation + +FIXTURES = Path(__file__).parent / "fixtures" +CORPUS = FIXTURES / "ceiling_corpus.jsonl.gz" +WATCHES = FIXTURES / "ceiling_watches.yaml" +EXAMPLE_POSITION = Path(__file__).resolve().parents[2] / "config" / "position.example.yaml" + +TODAY = date(2026, 8, 31) + +# The gate. See the module docstring for how it was measured and what it can and +# cannot see. Lowering it is a tightening and is welcome; raising it is a claim +# that Tier 1 should hand the adjudicator more work, and needs the same kind of +# measurement that produced it. +CEILING = 20 + +# The baseline as measured on 2026-08-31. Pinned separately from the ceiling: the +# ceiling is the gate, this is the tripwire, and a change in either direction is +# worth seeing rather than absorbing into the headroom. +MEASURED_ROUTED = 5 +MEASURED_LINKS = 5 + + +def _parse_published(value): + if not value: + return None + for fmt in ("%Y-%m-%d %H:%M:%S.%f", "%Y-%m-%d %H:%M:%S"): + try: + return datetime.strptime(value, fmt) + except ValueError: + continue + return None + + +class _WatchRow: + def __init__(self, watch): + self.id = watch.id + self.triggers = watch.triggers_json() + self.expires = watch.expires + + +@pytest.fixture(scope="module") +def corpus_rows(): + with gzip.open(CORPUS, "rt", encoding="utf-8") as handle: + return [json.loads(line) for line in handle] + + +@pytest.fixture(scope="module") +def ceiling_watches(): + position = load_position(EXAMPLE_POSITION, today=TODAY) + return load_watches(WATCHES, position=position, today=TODAY) + + +@pytest.fixture(scope="module") +def routed(corpus_rows, ceiling_watches): + """ + The whole corpus stored as real observations, then routed. Built once. + + Module-scoped because storing 1,000 observations computes 1,000 MinHash + signatures, and doing that per test would make the suite slow for no extra + confidence. + """ + engine = create_engine("sqlite://") + Base.metadata.create_all(bind=engine) + session = sessionmaker(bind=engine)() + + feeds: dict[str, RSSFeed] = {} + for index, row in enumerate(corpus_rows): + key = row["source_url"] or row["source_name"] + if key not in feeds: + feed = RSSFeed( + url=row["source_url"] or f"about:blank#{row['source_name']}", + name=row["source_name"] or "unknown", + ) + session.add(feed) + session.flush() + feeds[key] = feed + store_observation( + session, + feeds[key], + RawItem( + guid=row["guid"] or f"row-{index}", + url=row["url"], + title=row["title"], + normalized_content=row["body"], + published_date=_parse_published(row["published_date"]), + language=row["language"], + ), + ) + session.flush() + + report = route(session, today=TODAY, watches=[_WatchRow(watch) for watch in ceiling_watches]) + yield report + session.close() + + +class TestTheFixtureIsWhatTheCeilingWasMeasuredAgainst: + """ + If the corpus or the watch set drifts, K is a number about something else. + + A ceiling measured against one corpus and asserted against another is not a + gate, it is a coincidence. + """ + + def test_the_corpus_is_one_thousand_rows(self, corpus_rows): + assert len(corpus_rows) == 1000 + assert len({row["guid"] for row in corpus_rows}) == 1000 + + def test_the_corpus_is_real_article_text(self, corpus_rows): + """ + Real, not synthesised: 86 distinct feeds and bodies of realistic length. + + Synthetic strings written by whoever wrote the test would make K a + number about the test author's imagination. + """ + assert len({row["source_name"] for row in corpus_rows}) == 86 + bodies = [len(row["body"]) for row in corpus_rows] + assert sum(bodies) // len(bodies) == 896 + + def test_there_are_five_watches(self, ceiling_watches): + assert [w.id for w in ceiling_watches] == [ + "fedramp-conmon-scope-expands", + "fedramp-20x-supersedes-rev5", + "compliance-hiring-market-tightens", + "cisa-kev-cadence-holds", + "nist-800-53-revision-lands", + ] + + def test_every_observation_was_stored_and_considered(self, routed): + assert routed.considered == 1000 + + +class TestTheCeiling: + def test_fewer_than_k_observations_route(self, routed): + assert routed.routed_count < CEILING, ( + f"{routed.routed_count} of {routed.considered} observations routed, ceiling is " + f"{CEILING}. Tier 1 is the cost control and every routed observation is one " + f"adjudication. Per watch: {routed.per_watch}. Per clause: {routed.per_clause}." + ) + + def test_fewer_than_k_links_are_produced(self, routed): + """ + Links, not routed observations, are what the adjudicator is billed for. + + One observation routed to three watches is three adjudications. The two + numbers diverge exactly when a term is added to several watches at once, + which is why both are asserted. + """ + assert len(routed.links) < CEILING, ( + f"{len(routed.links)} links from {routed.considered} observations, ceiling is " + f"{CEILING}. Per clause: {routed.per_clause}." + ) + + def test_the_measured_baseline_has_not_moved(self, routed): + """ + The tripwire. Any change to selectivity shows up here before the gate. + + If this fails and the ceiling test still passes, routing changed within + the headroom -- which may be entirely correct, and should be re-measured + and re-recorded rather than absorbed. + """ + assert (routed.routed_count, len(routed.links)) == (MEASURED_ROUTED, MEASURED_LINKS) + + def test_almost_everything_dies_here(self, routed): + """The architectural claim, in the direction it is claimed.""" + assert routed.unrouted_count == routed.considered - routed.routed_count + assert routed.unrouted_count / routed.considered > 0.95 + + def test_the_unrouted_are_reported_as_hashes_not_as_a_number(self, routed): + """A bare integer is not a coverage-gap signal. See src/routing/gaps.py.""" + assert len(routed.unrouted) == routed.unrouted_count + assert all(h.startswith("sha256:") for h in routed.unrouted) + + +class TestWideningIsAttributable: + def test_every_link_names_the_clause_that_produced_it(self, routed): + """ + "Watch X routed 400" is a bill. "Clause 2 of watch X routed 400" is a diff. + + Without the clause index there is no way to tell which line of the + watch file to edit. + """ + assert sum(routed.per_clause.values()) == len(routed.links) + for (watch_id, clause_index), count in routed.per_clause.items(): + assert watch_id in routed.per_watch + assert clause_index >= 0 + assert count > 0 diff --git a/tests/routing/test_gaps.py b/tests/routing/test_gaps.py new file mode 100644 index 0000000..823add2 --- /dev/null +++ b/tests/routing/test_gaps.py @@ -0,0 +1,149 @@ +""" +The coverage-gap signal: what matched nothing, grouped so it can be read. + +The landmine this defends: an unrouted *count* is the only place a missing +sensor is visible before a staleness alert fires, and a bare integer is not a +signal anyone can act on. These tests assert the count is accompanied by +something with a subject attached to it. +""" + +from __future__ import annotations + +import json + +from src.routing import GAP_REPORT_SCHEMA, cluster_unrouted, gap_report, gap_terms, route +from src.routing.gaps import default_gap_report_path, write_gap_report + +from .conftest import TODAY, WatchRow + +CISA = WatchRow("cisa-advisories", [{"entities": ["CISA"], "terms": ["advisory"]}]) + +WIRE = ( + "The county board voted on Tuesday to approve the stormwater levy after a " + "two-hour hearing, with the measure passing four votes to one." +) + + +class TestGapTerms: + def test_counts_documents_not_occurrences(self): + """One verbose article must not be able to invent a theme.""" + assert gap_terms(["levy levy levy levy", "levy hearing"]) == [ + ("levy", 2), + ("hearing", 1), + ] + + def test_drops_structural_words_and_short_tokens(self): + assert gap_terms(["The board said that it was a stormwater levy"]) == [ + ("board", 1), + ("levy", 1), + ("stormwater", 1), + ] + + def test_ties_break_alphabetically_so_the_report_is_deterministic(self): + assert gap_terms(["levy hearing board"]) == [("board", 1), ("hearing", 1), ("levy", 1)] + + def test_is_case_folded(self): + assert gap_terms(["Stormwater", "stormwater"]) == [("stormwater", 2)] + + +class TestClusters: + def test_near_duplicates_land_in_one_cluster(self, test_session, observe, feed): + """ + The same wire story from two outlets, seen by no watch, is one gap. + + This is the strongest form of the signal: the environment thought the + story mattered enough to carry twice and the operator has no sensor on + it. + """ + other = feed("Cardinal News", "https://cardinalnews.test/rss") + observe("County approves stormwater levy", WIRE) + observe("County approves stormwater levy", WIRE + " Reporting by staff.", source=other) + + report = route(test_session, today=TODAY, watches=[CISA]) + assert report.unrouted_count == 2 + + clusters = cluster_unrouted(test_session, report.unrouted) + assert [c.size for c in clusters] == [2] + assert clusters[0].representative_title == "County approves stormwater levy" + assert "stormwater" in dict(clusters[0].terms) + assert set(clusters[0].sources) == {"FedScoop", "Cardinal News"} + + def test_unrelated_observations_stay_separate(self, test_session, observe): + observe("County approves stormwater levy", WIRE) + observe("Ferry service resumes", "The harbour authority restored the summer timetable.") + + report = route(test_session, today=TODAY, watches=[CISA]) + clusters = cluster_unrouted(test_session, report.unrouted) + assert [c.size for c in clusters] == [1, 1] + + def test_largest_cluster_first(self, test_session, observe, feed): + second = feed("Cardinal News", "https://cardinalnews.test/rss") + observe("County approves stormwater levy", WIRE) + observe("County approves stormwater levy", WIRE + " Reporting by staff.", source=second) + observe("Ferry service resumes", "The harbour authority restored the summer timetable.") + + report = route(test_session, today=TODAY, watches=[CISA]) + assert [c.size for c in cluster_unrouted(test_session, report.unrouted)] == [2, 1] + + def test_a_routed_observation_is_not_in_any_cluster(self, test_session, observe): + routed = observe("CISA issues an advisory", "CISA published an advisory today") + observe("County approves stormwater levy", WIRE) + + report = route(test_session, today=TODAY, watches=[CISA]) + members = {h for c in cluster_unrouted(test_session, report.unrouted) for h in c.members} + assert routed not in members + assert len(members) == 1 + + def test_no_unrouted_observations_means_no_clusters(self, test_session, observe): + observe("CISA issues an advisory", "CISA published an advisory today") + report = route(test_session, today=TODAY, watches=[CISA]) + assert cluster_unrouted(test_session, report.unrouted) == [] + + +class TestTheReportTask021Reads: + """ + The output is a file with a schema, at a path, holding subjects. + + Task 021 proposes watches from these clusters. It is not written yet, so + what this task owes it is a stable shape and a stable place -- and the + schema string so it can refuse a report it does not understand rather than + read a field that moved. + """ + + def test_the_report_carries_counts_clusters_and_terms(self, test_session, observe): + observe("CISA issues an advisory", "CISA published an advisory today") + observe("County approves stormwater levy", WIRE) + + report = route(test_session, today=TODAY, watches=[CISA]) + payload = gap_report(test_session, report) + + assert payload["schema"] == GAP_REPORT_SCHEMA + assert payload["observations_considered"] == 2 + assert payload["routed_observations"] == 1 + assert payload["unrouted_observations"] == 1 + assert payload["watches"] == ["cisa-advisories"] + assert payload["per_watch"] == {"cisa-advisories": 1} + assert len(payload["clusters"]) == 1 + assert "stormwater" in dict(tuple(t) for t in payload["gap_terms"]) + + def test_the_unrouted_count_never_travels_without_its_clusters(self, test_session, observe): + """The whole point. A count with nothing attached is not a signal.""" + observe("County approves stormwater levy", WIRE) + payload = gap_report(test_session, route(test_session, today=TODAY, watches=[CISA])) + + assert payload["unrouted_observations"] > 0 + assert payload["clusters"] + assert payload["gap_terms"] + + def test_it_is_written_as_json_where_it_is_asked_for(self, test_session, observe, tmp_path): + observe("County approves stormwater levy", WIRE) + payload = gap_report(test_session, route(test_session, today=TODAY, watches=[CISA])) + + target = tmp_path / "nested" / "unrouted.json" + assert write_gap_report(payload, target) == target + assert json.loads(target.read_text())["schema"] == GAP_REPORT_SCHEMA + + def test_the_default_path_is_under_the_data_directory(self): + path = default_gap_report_path() + assert path.name == "unrouted_clusters.json" + assert path.parent.name == "routing" diff --git a/tests/routing/test_no_model.py b/tests/routing/test_no_model.py new file mode 100644 index 0000000..2c0aa67 --- /dev/null +++ b/tests/routing/test_no_model.py @@ -0,0 +1,247 @@ +""" +Proof that no model is reachable from Tier 1. + +Tier 1 is the cost control. The claim it rests on -- "most observations match +nothing and die here" -- is only worth anything if dying here is free, which +means nothing in this tier may call a model, directly or through three layers of +convenience import. + +Two kinds of proof, because either alone is weak: + +* :class:`TestRoutingNeedsNoAnthropicClient` runs the **full routing path** in a + subprocess in which ``anthropic`` and ``src.llm`` cannot be imported at all -- + any attempt raises ``ImportError`` from a meta-path hook installed before a + single line of ``src`` is loaded -- and with ``ANTHROPIC_API_KEY`` removed from + the environment. Ingestion, compilation, routing, persistence, clustering and + writing the gap report all complete. This is the behavioural proof, and it is + a subprocess because an in-process one would be testing modules Python had + already imported. +* :class:`TestTheSourceTreeHasNoModelImport` reads the source. The behavioural + test only exercises the paths it happens to take; a model call on a branch + behind a flag would survive it. + +Added 2026-08-31 for backlog task 015. +""" + +from __future__ import annotations + +import ast +import os +import subprocess +import sys +from pathlib import Path + +ROOT = Path(__file__).resolve().parents[2] +ROUTING = ROOT / "src" / "routing" + +# Driven in a subprocess. The blocker is installed at line one, before src is +# importable, so nothing can have been imported already. +DRIVER = r""" +import sys + +BANNED = ("anthropic", "src.llm") + +class NoModelsHere: + def find_module(self, name, path=None): + return self.find_spec(name, path) + + def find_spec(self, name, path=None, target=None): + if name == "anthropic" or name.startswith("anthropic.") or name.startswith("src.llm"): + raise ImportError( + f"refusing to import {name}: Tier 1 routing must reach no model. " + f"If this fired, a model call became reachable from src/routing." + ) + return None + +sys.meta_path.insert(0, NoModelsHere()) + +for name in list(sys.modules): + if name == "anthropic" or name.startswith("anthropic.") or name.startswith("src."): + del sys.modules[name] + +from datetime import date, datetime + +from sqlalchemy import create_engine +from sqlalchemy.orm import sessionmaker + +from src.database.models import Base, RouteCandidate, RSSFeed +from src.routing import compile_watch, gap_report, persist, route, write_gap_report +from src.sources.base import RawItem +from src.sources.observation import store_observation + +assert not any(n == "anthropic" or n.startswith("src.llm") for n in sys.modules), sys.modules.keys() + +engine = create_engine("sqlite://") +Base.metadata.create_all(bind=engine) +db = sessionmaker(bind=engine)() + +feed = RSSFeed(url="https://example.test/rss", name="Federal Register") +db.add(feed) +db.flush() + +texts = [ + ("CISA issues an advisory", "CISA published an advisory about a known exploited flaw."), + ("County approves stormwater levy", "The county board voted on Tuesday to approve the levy."), + ("A FedRAMP notice", "GSA posted a FedRAMP continuous monitoring notice."), +] +for i, (title, body) in enumerate(texts): + store_observation( + db, + feed, + RawItem( + guid=f"g{i}", + url=f"https://example.test/{i}", + title=title, + normalized_content=body, + published_date=datetime(2026, 8, 20), + ), + ) +db.flush() + + +class W: + def __init__(self, i, t): + self.id, self.triggers, self.expires = i, t, date(2027, 1, 1) + + +watches = [ + W("cisa", [{"entities": ["CISA"], "terms": ["advisory"]}]), + W("fedramp", [{"entities": ["GSA"], "terms": ["continuous monitoring"]}]), + W("register", [{"sources": ["Federal Register"], "terms": ["FedRAMP"]}]), +] + +compile_watch("cisa", watches[0].triggers) +report = route(db, today=date(2026, 8, 31), watches=watches) +written = persist(db, report) +payload = gap_report(db, report) +import tempfile, pathlib +write_gap_report(payload, pathlib.Path(tempfile.mkdtemp()) / "gaps.json") + +assert report.considered == 3, report.considered +assert report.routed_count == 2, report.routed_count +assert report.unrouted_count == 1, report.unrouted_count +assert written["inserted"] == 3, written +assert db.query(RouteCandidate).count() == 3 +assert payload["clusters"], payload +print("ROUTED_WITHOUT_A_MODEL", report.considered, report.routed_count, len(report.links)) +""" + + +class TestRoutingNeedsNoAnthropicClient: + def test_the_full_path_runs_with_the_client_unimportable(self): + env = {k: v for k, v in os.environ.items() if k != "ANTHROPIC_API_KEY"} + env["PYTHONPATH"] = str(ROOT) + env["DATABASE_URL"] = "sqlite://" + + result = subprocess.run( + [sys.executable, "-c", DRIVER], + cwd=ROOT, + env=env, + capture_output=True, + text=True, + timeout=120, + ) + + assert result.returncode == 0, ( + f"routing could not complete without the Anthropic client.\n" + f"stdout:\n{result.stdout}\nstderr:\n{result.stderr}" + ) + assert "ROUTED_WITHOUT_A_MODEL 3 2 3" in result.stdout, result.stdout + + def test_the_blocker_in_that_subprocess_actually_blocks(self): + """ + The control. Without it the test above proves only that the code ran. + + If ``import anthropic`` succeeds under the same hook, the hook is + inert and the passing test above means nothing. + """ + env = {k: v for k, v in os.environ.items() if k != "ANTHROPIC_API_KEY"} + env["PYTHONPATH"] = str(ROOT) + + probe = DRIVER.split("from datetime import")[0] + ( + "\ntry:\n" + " import anthropic\n" + "except ImportError as exc:\n" + " print('BLOCKED', exc)\n" + "else:\n" + " raise SystemExit('the import hook did not block anthropic')\n" + ) + result = subprocess.run( + [sys.executable, "-c", probe], + cwd=ROOT, + env=env, + capture_output=True, + text=True, + timeout=60, + ) + assert result.returncode == 0, result.stderr + assert result.stdout.startswith("BLOCKED"), result.stdout + + +class TestTheSourceTreeHasNoModelImport: + """ + Read the source, not just the paths a test happens to take. + + A model call behind a feature flag, in an except branch, or in a function no + test calls would pass the behavioural test above and still bill. + """ + + def _sources(self): + return sorted(ROUTING.rglob("*.py")) + [ROOT / "src" / "cli" / "route.py"] + + @staticmethod + def _code(path: Path) -> tuple[set[str], set[str]]: + """ + ``(imported module paths, names used as code)`` for one file. + + Parsed rather than grepped. A grep would be satisfied by the sentence + "this module imports no Anthropic client", which is the sort of comment + that gets written and then stops being true. + """ + tree = ast.parse(path.read_text(encoding="utf-8")) + modules: set[str] = set() + names: set[str] = set() + for node in ast.walk(tree): + if isinstance(node, ast.Import): + modules.update(alias.name for alias in node.names) + elif isinstance(node, ast.ImportFrom): + modules.add(node.module or "") + names.update(alias.name for alias in node.names) + elif isinstance(node, ast.Name): + names.add(node.id) + elif isinstance(node, ast.Attribute): + names.add(node.attr) + return modules, names + + def test_no_routing_module_imports_a_model_client(self): + offenders = [] + for path in self._sources(): + modules, names = self._code(path) + for module in modules: + tail = module.lstrip(".") + if tail.startswith("anthropic") or tail.startswith("llm") or ".llm" in tail: + offenders.append(f"{path.relative_to(ROOT)} imports {module!r}") + for banned in ("ClaudeClient", "AsyncAnthropic", "Anthropic"): + if banned in names: + offenders.append(f"{path.relative_to(ROOT)} uses {banned!r}") + assert offenders == [], ( + f"Tier 1 must reach no model: {offenders}. Routing decides candidacy with " + f"regexes; the tier that reasons is Tier 2, and it is the only one that costs " + f"money per item." + ) + + def test_no_routing_module_reads_the_articles_table(self): + """ + Observations are authoritative; ``articles`` is the pre-rewrite archive. + + The rule is written in src/database/models.py. This asserts the package + obeys it rather than trusting that it does -- and it reads the parse + tree, so the router's docstring saying it does not touch Article is not + what makes this pass. + """ + offenders = [ + str(path.relative_to(ROOT)) + for path in self._sources() + if "Article" in self._code(path)[1] + ] + assert offenders == [], f"these routing modules use Article: {offenders}" diff --git a/tests/routing/test_predicate.py b/tests/routing/test_predicate.py new file mode 100644 index 0000000..bd27d93 --- /dev/null +++ b/tests/routing/test_predicate.py @@ -0,0 +1,261 @@ +""" +Compiling triggers into a predicate: the shape, the refusals, and the boundaries. + +**On the boundary class in this file, and why it is written the way it is.** +Backlog task 010 found that its own boundary tests passed for the wrong reason: +every collision case used a SHOUTED term, and a shouted term matches +case-sensitively, so ``combat`` was rejected as a mention of ``OMB`` before the +anchors were ever consulted. The whole class passed with ``LEFT_BOUNDARY`` and +``RIGHT_BOUNDARY`` deleted from the codebase. + +So every case in :class:`TestWordBoundariesAreLoadBearing` uses a term that is +**not** shouted -- ``nist``, ``mail``, ``ai``, ``rev`` -- and is therefore +compiled case-insensitively. Case cannot reject any of them. The anchors are the +only thing standing between the term and its host word, which means deleting the +anchors makes the class fail. That was checked by deleting them, on 2026-08-31; +see the class docstring for what happened. +""" + +from __future__ import annotations + +import pytest + +from src.routing import CompiledWatch, TriggerCompileError, compile_watch, source_keys + + +def watch(*clauses, watch_id="w") -> CompiledWatch: + return compile_watch(watch_id, list(clauses)) + + +def fires(compiled, text, *sources) -> bool: + return compiled.matches(text, source_keys(*sources)) + + +class TestClauseSemantics: + """AND within a clause, OR across clauses. This is the whole grammar.""" + + def test_terms_within_a_field_are_or(self): + w = watch({"terms": ["ConMon", "continuous monitoring"]}) + assert fires(w, "the ConMon burden grew") + assert fires(w, "continuous monitoring requirements") + assert not fires(w, "an unrelated rule") + + def test_fields_within_a_clause_are_and(self): + w = watch({"entities": ["GSA"], "terms": ["continuous monitoring"]}) + assert fires(w, "GSA expanded continuous monitoring") + assert not fires(w, "GSA expanded nothing in particular") + assert not fires(w, "continuous monitoring, by someone else entirely") + + def test_clauses_are_or(self): + w = watch( + {"entities": ["GSA"], "terms": ["continuous monitoring"]}, + {"terms": ["FedRAMP 20x"]}, + ) + assert fires(w, "GSA expanded continuous monitoring") + assert fires(w, "FedRAMP 20x lands") + assert not fires(w, "neither of those things") + + def test_the_firing_clause_is_reported_by_index(self): + w = watch({"terms": ["alpha"]}, {"terms": ["beta"]}) + assert w.matching_clause("beta only", frozenset()) == 1 + assert w.matching_clause("alpha only", frozenset()) == 0 + assert w.matching_clause("nothing", frozenset()) is None + + def test_the_first_matching_clause_wins(self): + """Two clauses matching is still one candidate, attributed to the first.""" + w = watch({"terms": ["alpha"]}, {"terms": ["beta"]}) + assert w.matching_clause("alpha and beta", frozenset()) == 0 + + +class TestSourceAllowlist: + """ + ``sources`` is an allowlist matched by equality, not by substring. + + Substring matching here would mean a clause naming "Federal Register" also + accepting a blog called "Federal Register Watch", which is the same class of + error as an unanchored term and is worse, because it silently widens a + clause the operator wrote to be narrow. + + **A consequence, measured on the real corpus on 2026-08-31 and recorded + because it is a live question rather than a defect:** the feeds in this + repository are named "Federal Register", "Federal Register - Public + Inspection" and "Federal Register - Documents API", and only the last of + those has stored articles. A clause saying ``sources: [Federal Register]`` + therefore matches none of them. Equality is the conservative reading -- it + can only under-route, never over-route, and under-routing surfaces as an + unrouted cluster. Whether an operator means the exact feed or the family of + feeds is a question for the operator, not an assumption to make here. + """ + + def test_matches_the_feed_name(self): + w = watch({"sources": ["Federal Register"], "terms": ["FedRAMP"]}) + assert fires(w, "a FedRAMP notice", "Federal Register") + + def test_matches_the_feed_url(self): + w = watch({"sources": ["https://www.federalregister.gov/articles.rss"]}) + assert fires( + w, "anything", "Federal Register", "https://www.federalregister.gov/articles.rss" + ) + + def test_is_case_and_whitespace_insensitive(self): + w = watch({"sources": [" federal REGISTER "]}) + assert fires(w, "anything", "Federal Register") + + def test_does_not_match_a_longer_name(self): + w = watch({"sources": ["Federal Register"]}) + assert not fires(w, "anything", "Federal Register Watch Blog") + + def test_a_source_clause_still_ands_with_its_terms(self): + w = watch({"sources": ["Federal Register"], "terms": ["FedRAMP"]}) + assert not fires(w, "an unrelated notice", "Federal Register") + assert not fires(w, "a FedRAMP notice", "FedScoop") + + +class TestWordBoundariesAreLoadBearing: + """ + Terms match whole words. Every case here uses a non-shouted term. + + **Verified by deletion on 2026-08-31.** With ``LEFT_BOUNDARY`` and + ``RIGHT_BOUNDARY`` set to the empty string in + ``src/matching/entity_matcher.py`` and ``term_pattern`` reduced to the bare + escaped term, every test in this class fails. That is the check task 010's + equivalent class did not survive: its cases were shouted acronyms, rejected + by the case rule before the anchors were reached, so it passed with the + anchors gone. + + The scale being defended, measured on this repository's 55,249-article + corpus: ``nist`` appears as a substring in 5,364 titles and at a word + boundary in 73; ``mail`` is 1,842 against 339; the whole-text substring + count for ``opm`` is 1,706 -- almost all of it the word "development" -- + against 14 at a boundary. + """ + + @pytest.mark.parametrize( + "host", + [ + "The administration said so.", + "the minister of the interior", + "Afghanistan's economy", + "a sinister development", + "communist party congress", + ], + ) + def test_lowercase_nist_does_not_match_its_host_words(self, host): + """ + ``nist`` is lowercase, so it compiles case-insensitively. + + There is no case rule available to reject "admiNISTration" here. If the + anchors go, this passes and the watch starts routing 5,364 titles. + """ + w = watch({"terms": ["nist"]}) + assert not fires(w, host) + + def test_lowercase_nist_does_match_the_whole_word(self): + w = watch({"terms": ["nist"]}) + assert fires(w, "the nist catalog") + assert fires(w, "NIST published a revision") + + @pytest.mark.parametrize( + "host", ["send an email", "blackmail allegations", "a mailbox rule", "mailing list"] + ) + def test_mail_does_not_match_its_host_words(self, host): + w = watch({"terms": ["mail"]}) + assert not fires(w, host) + + def test_mail_does_match_the_whole_word(self): + w = watch({"terms": ["mail"]}) + assert fires(w, "the mail was delayed") + assert fires(w, "delivered by MAIL yesterday") + + @pytest.mark.parametrize( + "host", ["he said nothing", "maintain the system", "a certain claim", "chairman"] + ) + def test_two_letter_ai_does_not_match_its_host_words(self, host): + w = watch({"terms": ["ai"]}) + assert not fires(w, host) + + def test_a_multiword_term_is_anchored_at_both_ends(self): + w = watch({"terms": ["rev 5"]}) + assert not fires(w, "prerev 5 draft") + assert not fires(w, "rev 55 of the document") + assert fires(w, "Rev 5 baseline") + + def test_a_hyphen_and_a_slash_are_boundaries_not_word_characters(self): + """ "CISA-issued" is a mention; "precisa" is not. Both matter.""" + w = watch({"terms": ["conmon"]}) + assert fires(w, "conmon-related work") + assert fires(w, "conmon/significant-change reporting") + assert not fires(w, "preconmonitor readings") + + def test_a_term_split_across_a_line_wrap_still_matches(self): + w = watch({"terms": ["continuous monitoring"]}) + assert fires(w, "expanded continuous\n monitoring duties") + + +class TestShoutedAcronymsAreCaseSensitive: + """ + The second rule, tested separately so neither can cover for the other. + + These cases would pass with the anchors deleted -- that is the point of + keeping them out of the boundary class. + """ + + def test_a_shouted_term_is_not_matched_in_lower_case(self): + w = watch({"entities": ["OMB"]}) + assert fires(w, "OMB issued guidance") + assert not fires(w, "the omb issued guidance") + + def test_an_unshouted_term_is_matched_in_any_case(self): + w = watch({"entities": ["FedRAMP PMO"]}) + assert fires(w, "the fedramp pmo said") + assert fires(w, "the FEDRAMP PMO said") + + +class TestRefusals: + """ + Everything this compiler cannot evaluate, it refuses. It never skips. + + A skipped clause is a watch that has quietly stopped watching; a kept empty + clause is a watch that matches the entire corpus. The loader already rejects + both, but the loader is not the only way JSON reaches ``watches.triggers`` -- + a hand-edited row, a restored dump and a future migration all bypass it. + """ + + def test_a_clause_constraining_nothing_is_refused(self): + with pytest.raises(TriggerCompileError, match="constrains nothing"): + compile_watch("w", [{}]) + + def test_a_clause_whose_fields_are_all_blank_is_refused(self): + with pytest.raises(TriggerCompileError, match="every entry is blank"): + compile_watch("w", [{"terms": [" ", ""]}]) + + def test_empty_triggers_are_refused(self): + with pytest.raises(TriggerCompileError, match="non-empty list"): + compile_watch("w", []) + + def test_prose_triggers_are_refused(self): + with pytest.raises(TriggerCompileError, match="non-empty list"): + compile_watch("w", "anything about FedRAMP") + + def test_a_prose_clause_is_refused(self): + with pytest.raises(TriggerCompileError, match="expected a mapping"): + compile_watch("w", ["anything about FedRAMP"]) + + def test_a_string_where_a_list_belongs_is_refused(self): + with pytest.raises(TriggerCompileError, match="must be a list of strings"): + compile_watch("w", [{"terms": "FedRAMP"}]) + + def test_an_unknown_clause_field_is_refused(self): + with pytest.raises(TriggerCompileError, match="unknown field"): + compile_watch("w", [{"terms": ["FedRAMP"], "authors": ["someone"]}]) + + def test_a_clause_that_would_lose_a_conjunct_is_refused(self): + """ + A declared field whose every value is blank would silently widen the clause. + + ``{"entities": [" "], "terms": ["FedRAMP"]}`` reads as "FedRAMP AND some + entity" and would evaluate as "FedRAMP", which is a different and + broader watch than the one written down. + """ + with pytest.raises(TriggerCompileError, match="every entry is blank"): + compile_watch("w", [{"entities": [" "], "terms": ["FedRAMP"]}]) diff --git a/tests/routing/test_router.py b/tests/routing/test_router.py new file mode 100644 index 0000000..7d32755 --- /dev/null +++ b/tests/routing/test_router.py @@ -0,0 +1,173 @@ +""" +Running the predicates over stored observations, and recording what matched. + +The properties here are the ones that cost money if they break: what is +considered, what is written, and that writing it twice writes it once. +""" + +from __future__ import annotations + +from datetime import date + +import pytest + +from src.database.models import RouteCandidate +from src.routing import persist, route + +from .conftest import TODAY, WatchRow + +CISA = WatchRow("cisa-advisories", [{"entities": ["CISA"], "terms": ["advisory", "alert"]}]) +FEDRAMP = WatchRow("fedramp-conmon", [{"terms": ["FedRAMP"], "entities": ["GSA"]}]) + + +class TestWhatRoutes: + def test_an_observation_matching_nothing_is_unrouted(self, test_session, observe): + h = observe("A local school board meeting", "nothing to do with any watch") + report = route(test_session, today=TODAY, watches=[CISA, FEDRAMP]) + + assert report.considered == 1 + assert report.routed_count == 0 + assert report.unrouted == [h] + assert report.per_watch == {"cisa-advisories": 0, "fedramp-conmon": 0} + + def test_a_matching_observation_is_linked_to_the_watch_that_matched( + self, test_session, observe + ): + observe("CISA issues an advisory", "CISA published an advisory today") + report = route(test_session, today=TODAY, watches=[CISA, FEDRAMP]) + + assert report.routed_count == 1 + assert report.unrouted == [] + assert [(link.watch_id, link.clause_index) for link in report.links] == [ + ("cisa-advisories", 0) + ] + + def test_one_observation_can_route_to_two_watches(self, test_session, observe): + observe("CISA advisory names GSA", "CISA published an advisory naming GSA and FedRAMP") + report = route(test_session, today=TODAY, watches=[CISA, FEDRAMP]) + + assert report.routed_count == 1 + assert len(report.links) == 2 + assert report.fan_out == 2.0 + + def test_fan_out_is_links_per_observation_not_routed_observations(self, test_session, observe): + observe("CISA advisory names GSA", "CISA published an advisory naming GSA and FedRAMP") + observe("A local school board meeting", "nothing relevant") + report = route(test_session, today=TODAY, watches=[CISA, FEDRAMP]) + + assert (report.considered, report.routed_count, len(report.links)) == (2, 1, 2) + assert report.fan_out == 1.0 + + def test_the_source_axis_is_read_from_the_feed_row(self, test_session, observe, feed): + register = feed("Federal Register", "https://federalregister.gov/rss") + watch = WatchRow("fr-only", [{"sources": ["Federal Register"], "terms": ["FedRAMP"]}]) + + observe("A FedRAMP notice", "text", source=register) + observe("A FedRAMP story", "text") # default feed, FedScoop + + report = route(test_session, today=TODAY, watches=[watch]) + assert report.per_watch == {"fr-only": 1} + + def test_limit_considers_only_the_last_n(self, test_session, observe): + for i in range(5): + observe(f"item {i}", "nothing relevant") + + assert route(test_session, today=TODAY, watches=[CISA], limit=2).considered == 2 + assert route(test_session, today=TODAY, watches=[CISA]).considered == 5 + + +class TestExpiredWatchesAreNotRouted: + """ + Volume scales with *live* watches, so an expired one is not one. + + Routing to an expired watch spends the adjudicator's budget on a question + that stopped mattering. The skip is reported rather than silent: "nothing + routed" and "every watch expired" must not look the same. + """ + + def test_an_expired_watch_is_skipped_and_named(self, test_session, observe): + observe("CISA issues an advisory", "CISA published an advisory today") + expired = WatchRow("old", [{"entities": ["CISA"]}], expires=date(2026, 1, 1)) + + report = route(test_session, today=TODAY, watches=[expired, CISA]) + + assert report.expired_watch_ids == ("old",) + assert report.watch_ids == ("cisa-advisories",) + assert report.per_watch == {"cisa-advisories": 1} + + def test_a_watch_expiring_today_is_still_live(self, test_session, observe): + observe("CISA issues an advisory", "CISA published an advisory today") + today_watch = WatchRow("today", [{"entities": ["CISA"]}], expires=TODAY) + + report = route(test_session, today=TODAY, watches=[today_watch]) + assert report.per_watch == {"today": 1} + + +class TestIdempotency: + """ + The same observation routed twice produces one link. + + Tier 1 is deterministic, so a second run over an unchanged corpus computes + the same links. Writing them again would double every watch's candidate + count and, downstream, adjudicate the same document twice. + """ + + def test_routing_twice_writes_one_row(self, test_session, observe): + observe("CISA issues an advisory", "CISA published an advisory today") + + first = persist(test_session, route(test_session, today=TODAY, watches=[CISA])) + second = persist(test_session, route(test_session, today=TODAY, watches=[CISA])) + + assert first == {"inserted": 1, "already_linked": 0} + assert second == {"inserted": 0, "already_linked": 1} + assert test_session.query(RouteCandidate).count() == 1 + + def test_a_third_run_is_still_a_no_op(self, test_session, observe): + observe("CISA issues an advisory", "CISA published an advisory today") + observe("CISA issues another advisory", "a second CISA advisory") + + for _ in range(3): + persist(test_session, route(test_session, today=TODAY, watches=[CISA])) + + assert test_session.query(RouteCandidate).count() == 2 + + def test_the_database_refuses_a_duplicate_written_around_persist(self, test_session, observe): + """ + The unique constraint, not just the pre-read, is what holds. + + ``persist`` skips what it already sees, which makes the *report* honest. + Two concurrent routing runs would both pass that check, so the guarantee + has to live in the schema as well. + """ + from sqlalchemy.exc import IntegrityError + + h = observe("CISA issues an advisory", "CISA published an advisory today") + persist(test_session, route(test_session, today=TODAY, watches=[CISA])) + + test_session.add( + RouteCandidate(observation_hash=h, watch_id="cisa-advisories", clause_index=0) + ) + with pytest.raises(IntegrityError): + test_session.flush() + test_session.rollback() + + def test_a_new_observation_adds_exactly_one_link(self, test_session, observe): + observe("CISA issues an advisory", "CISA published an advisory today") + persist(test_session, route(test_session, today=TODAY, watches=[CISA])) + + observe("CISA issues a later advisory", "another CISA advisory, later") + written = persist(test_session, route(test_session, today=TODAY, watches=[CISA])) + + assert written == {"inserted": 1, "already_linked": 1} + assert test_session.query(RouteCandidate).count() == 2 + + +class TestRouteWritesNothing: + """``route`` computes; ``persist`` writes. The dry run is not a mode of the writer.""" + + def test_route_alone_leaves_the_table_empty(self, test_session, observe): + observe("CISA issues an advisory", "CISA published an advisory today") + report = route(test_session, today=TODAY, watches=[CISA]) + + assert len(report.links) == 1 + assert test_session.query(RouteCandidate).count() == 0