Repository navigation
tier 1: deterministic routing - #24
Closed
sam-aydlette wants to merge 2 commits into
Closed
sam-aydlette wants to merge 2 commits into
sam-aydlette wants to merge 2 commits into
Conversation
Adjudication is the only tier that calls a model and it costs money per item. Tier 1 decides what it ever sees, so "notification volume scales with live watches rather than news volume" is a property of src/routing/, not a slogan. A predicate that is slightly too loose does not raise, does not log and does not fail a test that only checks it matches what it should. It bills. src/routing/predicate.py Compiles a watch's triggers -- a disjunction of conjunctions over terms, entities and a source allowlist -- into two regex alternations and a set membership test. Every pattern comes from entity_matcher.compile_terms, extracted from compile_entities for this task so there is one matcher and one set of anchors rather than two. Sources match by equality, not substring: "Federal Register" must not be satisfied by "Federal Register Watch Blog". It refuses rather than skips. A clause that constrains nothing matches the whole corpus; a clause whose declared field is all blanks silently loses a conjunct and becomes broader than it reads. The YAML loader already rejects both, but it is not the only way JSON reaches watches.triggers -- a hand-edited row, a restored dump, a migration written in a hurry. src/routing/router.py, route_candidates route() computes and writes nothing; persist() writes. The dry run is not a mode of the writer. A match becomes one row unique on (observation_hash, watch_id), so routing the same observation twice produces one link -- pre-read for an honest report, unique constraint because two concurrent runs would both pass the pre-read. The row records which clause fired: "watch X routed 400" is a bill, "clause 2 of watch X routed 400" is a diff. Expired watches are skipped and named, because "nothing routed" and "every watch expired last week" must not look the same. src/routing/gaps.py, salience.py The unrouted count is the coverage-gap signal -- the only place a missing sensor is visible before a staleness alert fires, and a bare integer makes it useless. Unrouted observations are grouped by their stored MinHash signatures (the same grouping the rest of the system uses, not a new algorithm) and labelled with the words they share; a document-frequency histogram catches the subject covered by forty different stories no two of which are near-duplicates. Written to data/routing/unrouted_clusters.json under a schema string. Task 021 reads it. This module proposes nothing -- invariant 6 is lost one convenient path at a time. insightweaver route [--dry-run] Per watch and per clause, how many of the last N observations route; then the routed and unrouted counts, the fan-out in adjudications per observation, the clusters, and where the report went. WORD BOUNDARIES, VERIFIED BY DELETION Task 010 found its own boundary tests passing for the wrong reason: every case used a shouted acronym, so the case rule rejected the collision before the anchors were consulted, and the class passed with LEFT_BOUNDARY and RIGHT_BOUNDARY deleted. Every case in TestWordBoundariesAreLoadBearing uses a NON-shouted term -- nist, mail, ai, rev 5 -- compiled case-insensitively, so the anchors are the only thing that can reject them. Emptying both anchors: 15 failed, 3 passed. Restored: 18 passed. The scale being defended, measured on this repository's corpus: nist is a substring of 5,364 titles and a boundary match in 73; the whole-text substring count for opm is 1,706, almost all of it the word "development", against 14. THE CEILING, AND HOW K WAS MEASURED tests/routing/test_ceiling.py: 1,000 observations against 5 watches route fewer than 20. The corpus is REAL article text. 1,000 rows were lifted from the 55,249-row pre-rewrite archive through a read-only (immutable=1) SQLite handle, stratified 400/600 by feed category -- deliberately not by trigger word, which would have defined the corpus by the answer -- and selected by BLAKE2b of (id, guid) so the sample does not depend on row order or a PRNG. The test stores them through the real store_observation path, so the observations it routes have real hashes, real payloads and real signatures. Nothing was written back to any database; this was a measurement, not a backfill. Measured baseline: 5 of 1,000 route, 5 links, fan-out 0.005. Every widening regression was then measured against the same fixture: boundaries deleted, 5; case rule deleted, 5; both, 6; clause AND became OR, 33; term and entity constraints dropped, 1,000. K = 20 sits at four times the baseline and below the cheapest regression it can see. It cannot see the boundary regressions, and the docstring says so rather than implying otherwise -- every surface form in a realistic compliance watch set is either a shouted acronym or a phrase with no common host word. Boundaries are gated by the predicate tests above. A separate assertion pins the measured baseline exactly. The ceiling is the gate; the baseline is the tripwire. NO MODEL, PROVED TWO WAYS A subprocess installs a meta-path hook that raises on any import of anthropic or src.llm before a line of src is loaded, drops ANTHROPIC_API_KEY, and drives the whole path: ingestion, compilation, routing, persistence, clustering and writing the report. A control asserts the hook actually blocks, without which the first test proves only that the code ran. A second class parses the routing package's ASTs -- a grep would be satisfied by a comment claiming no model is imported. Two live questions are written into the task file rather than assumed away: a sources clause matches a feed name exactly (so the shipped example's "Federal Register" clause matches none of this corpus's three Federal Register feeds), and trigger entities match only the surface form written, because the alias registry died with the beat files in task 012. make check exits 0; env -u ANTHROPIC_API_KEY pytest tests/ -q exits 0. 730 tests, 89 of them new. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01M9gY2miAUXijbFm25YnFcq
It said the live pipeline still fetches through fetch_all_active_feeds 'exactly as before'. That stopped being true in task 025, which closed both that function and fetch_and_store_feed because they wrote articles with no observation and Tier 1 routing cannot see those. Flagged by 025's IC, which was correctly forbidden from touching this file. Corrected here because 015 is the task that makes the claim consequential: this adapter is now the only way an RSS article enters the corpus.
Owner
Author
|
Superseded. This implemented backlog/015 under the tier-based architecture (grid-axis triggers, |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Implements
backlog/015. 730 tests (628 → 730),make checkexit 0.Tier 1 is where the rewrite's central property is enforced: notification volume scales with live watches, not news volume. A predicate that is too loose does not fail — it bills, because Tier 2 is the only LLM tier and runs on whatever this hands it.
K = 20, measured
1,000 observations built from real article text lifted from the frozen
articlesarchive through a read-onlyimmutable=1handle, stratified by feed category rather than by trigger word — stratifying by trigger word would define the corpus by the answer. Committed as a fixture so the test asserts against exactly what K was measured on.Five deliberate regressions place K:
K = 20 is 4× baseline and below the cheapest regression it can see. The reviewer independently reproduced all five numbers by monkeypatching and re-running the shipped fixture, rather than reading the table.
The author disclosed that this test cannot see its most expensive landmine
Deleting the anchors moves the ceiling by zero. That was verified against the shipped test, not just the harness, and written into the test's docstring rather than left in a report:
Every surface form in a realistic compliance watch set is either a shouted acronym (saved by the case rule) or a phrase with no common host word. The division: boundaries are gated by the predicate tests, AND/OR and match-all by the ceiling. A ceiling test presented as covering boundary regressions would have been a false guarantee, and nobody would have checked.
Boundary tests avoid the trap task 010 fell into
Every case uses a non-shouted term (
nist,mail,ai,rev 5) compiled case-insensitively, so the case rule cannot reject a collision before the anchors run. That is exactly how task 010's boundary class came to pass with anchoring deleted from the codebase.Verified by stripping the anchors: 15 fail, restored 18 pass.
git diffonentity_matcher.pyshows the anchors untouched in the commit.One matcher, not two
entity_matcherwas refactored to exposecompile_terms()/matches_any(), and routing calls them. The reviewer confirmed reuse by diff rather than docstring claim. Same forpayload_text()— one definition of "the text of an observation".No LLM is reachable, proved with a control
A subprocess installs a meta-path hook that raises on any
anthropicorsrc.llmimport beforesrcloads, drops the API key, and drives ingestion → compile → route → persist → cluster → write. A control test asserts the hook actually blocks — without it, the primary test proves only that code ran. A second class parses the routing package's ASTs, because a grep is satisfied by a comment saying "imports no model".Idempotent, run not described
Plus a test inserting a duplicate around
persistand asserting the DB raises. The pre-read makes the report honest; the unique constraint makes the table honest.Two semantics surfaced rather than assumed
sources:matches feed name by exact equality. Chosen because it can only under-route — which surfaces as an unrouted cluster — rather than over-route, which bills. If the operator means the family, it becomes a prefix match.entities:match only the surface form written. The alias registry died with the beat files in task 012, soFedRAMP PMOdoes not matchFedRAMP Program Management Office. Single change point:_entity_terms.Both are in the task file with their reasoning, not silently decided.
A finding worth more than the code
All three shipped example watches route zero against the entire 55,249-article corpus.
fedrampappears as a substring in 5 articles;cmmcin 6.That is not a routing defect — it is the coverage problem in a new place, and it is exactly what the unrouted report exists to surface. Real run over 1,000 observations: 987 clusters, top gap terms
security (180), trump (114), virginia (96), federal (90).The old briefing showed you something regardless of coverage. This architecture, working perfectly, shows you nothing. That is an argument for treating the Position repo and its watches as the real gate on v0.
Also in this PR
src/sources/rss_adapter.py's docstring claimed the live pipeline "still fetches feeds throughfetch_all_active_feedsexactly as before". That stopped being true in task 025. Flagged by 025's IC, which was correctly forbidden from touching the file; corrected here because 015 is what makes the claim consequential — this adapter is now the only way an RSS article enters the corpus.