Repository navigation
beats: test coverage against events that actually happened - #17
Merged
Merged
Conversation
…orecast --due The ledger held 33 predictions and had graded none. 25 were phrased "X would signal Y" -- interpretation rules, which cannot be wrong -- and only 3 carried a date, so 19 expired unjudged. The ledger was not empty for want of a resolver; it was empty because nothing gradeable had been written into it, and because every command was read-only: the model wrote the graph and the human read it. Four write commands, all local (no API key, no network call): questions add "..." --cadence 90d predict 23 "Yes -- slips" --by 2026-12-31 --confidence 0.7 forecast --due resolve 41 --outcome no --note "..." --by and --confidence are required and rejected at entry, not defaulted. A claim with no resolution date can never come due, so it can never be graded; a confidence with a default is a non-commitment in a different costume. Both refusals store nothing. A cadence is not a deadline. questions.cadence says how often a question is worth re-examining; predictions.due_by says when a specific claim resolves, so a question reviewed quarterly can hold a claim resolving in three weeks. forecast --due tests each question against its own interval, and stamps last_reviewed_at whether or not anything moved -- a quiet question that reappears daily trains the operator to skim. predictions.author (operator | model) splits track-record: a plain hit rate for the operator, model predictions under their own heading, never mixed. Nothing auto-resolves from coverage; the human resolves, and resolved_at is recorded separately from due_by so grading three months late stays visible. A resolved prediction is not editable. Schema changes are in a reversible migration, add_calibration_loop. The predictions half is a SQLite table rebuild (relaxing made_in_synthesis_id to nullable, which ALTER TABLE cannot do) with row counts asserted on both sides; verified carrying all 33 existing rows, backfilled to author='model'. The downgrade refuses rather than misattributing an operator claim. track-record's output changed shape, so its two existing assertions were updated to read the model heading. Interactive dispatch now splits arguments with shlex, without which a multi-word claim reached Click as four arguments.
… counts Task 005 asked whether a beat could reach its domain and answered with document counts -- 469 Federal Register documents narrowed to 24, 38 in seven days. Both real, both green, and neither able to detect that the beat was structurally unable to see a personnel change, because a personnel change is not a document. The first live brief then missed the reinstatement of the FedRAMP director. The corpus held 3 incidental FedRAMP mentions across 50,983 articles and none within two weeks. An article count measures whether ingestion is running. It does not measure whether ingestion reaches the domain. `insightweaver beat coverage NAME` measures the second thing: a beat declares known past events as `coverage_probes`, and the command reports per probe whether a feed the beat subscribes to carried a matching article, and which one. - `terms` must all appear; each `any_of` group needs one member. A probe resting on a single bare term is rejected by the loader -- `FedRAMP` alone is matched by an AWS region-launch post, and a probe that passes on generic terms manufactures the confidence this exists to remove. - Matching is deterministic word-boundary matching through the same primitives coverage entities use. No model call. Measured on the corpus: 1,842 articles contain the substring "mail", 339 contain it as a word; 5,364 contain "nist" against 73 as a word. - A trailing `*` marks a stem (`reinstat*`). Explicit rather than inferred, so the widening is visible to whoever has to trust the result. - A probe whose window holds no articles from the beat's sources is INCONCLUSIVE, with the reason distinguishing "predates the corpus" from "corpus has not reached it" from a gap. It stays in the denominator; a probe set that decays to the events still in retention is a green light that means nothing. - An unmatched probe is re-checked corpus-wide, so "nobody carried it" and "feeds you do not subscribe to carried it" are distinguishable. - Exits 1 on any unmatched probe, 2 when nothing was measured (no probes declared, or all inconclusive). Reads only; never writes. The shipped beat declares four probes. Against the current corpus two match, the FedRAMP reinstatement does not, and BOD 22-01 predates the corpus -- which is the honest measurement task 010 was written to get.
TestSubstringMatchingWouldLie carried this task's central claim -- that word-boundary matching is load-bearing -- but could not actually check it. Every collision in NIST_COLLISIONS is lowercase, and a shouted term like NIST compiles case-sensitively, so case-sensitivity rejected those headlines before the boundary anchors were ever consulted. Measured by stripping the anchors from term_pattern in place, leaving the case rule untouched: that class stays fully green, 12 passed. That is not academic. The shipped beat's probes use shouted terms -- OPM and CISA -- so a regression that dropped the anchoring would have left exactly the production term type unguarded. The new class holds case constant instead: term and headline are both all-caps, so the boundary anchor is the only thing left that can reject the match. OPM inside DOPMA and TOPMOST, BOD inside BODY, OMB inside BOMBING, CISA inside PRECISA. Each case pairs the negative assertion with a check that the probe's *other* required term is present, so a headline that merely lacked everything could not satisfy it, and with a positive control asserting the same probe still matches the term standing alone. Verified by the same method: with the anchors stripped from the production regex, 13 of the 17 fail and the 4 that pass are the positive controls, which should still match. No change to the matcher, the schema, the beat config, the CLI, or any existing test.
TestSubstringMatchingWouldLie said 'The boundary, not the two-level shape, is what rejects these.' That is false for its own cases: every term in it is shouted, so the compiled pattern is case-sensitive and the lowercase collisions are refused before the anchors are consulted. Deleting LEFT_BOUNDARY and RIGHT_BOUNDARY from entity_matcher leaves all 12 passing. The IC that found this correctly declined to edit an existing test outside its scope and flagged it instead. Correcting it here rather than leaving a demonstrably false claim next to the class that disproves it.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Implements
backlog/010-known-event-coverage-test.md.Why
Task 005 asked "can the beat reach its domain" and answered with document counts — 469 Federal Register documents narrowed to 24, 38 in seven days. Both real. Both green. Neither could detect that the beat was structurally unable to see a personnel change, because those are not documents. The first live brief then missed the reinstatement of the FedRAMP director.
Article count measures whether ingestion is running, not whether it reaches the domain. This is the test that replaces it:
insightweaver beat coverage NAMEA beat declares
coverage_probes— known past events plus the distinctive terms any report would carry. The command reports per probe, names which feed carried it, and exits non-zero when any probe is unmatched, so it can gate.That
elsewhereline is the point: it separates "nobody carried it" from "feeds you don't subscribe to carried it" — different problems with different fixes. Run against a corpus containing task 009's sources, it diagnoses the state exactly.The reviewer found the test guarding this was inert
TestSubstringMatchingWouldLiecarried the task's central claim — that word-boundary matching is load-bearing. Its cases are confounded by acronym case-sensitivity: every collision is lowercase and every term is shouted, so the case rule rejects them before the anchors are ever consulted. And the shipped beat's own probes use exactly those terms —OPM,CISA.I reproduced it by deleting
LEFT_BOUNDARYandRIGHT_BOUNDARYfrom production code:TestBoundariesAreLoadBearingForShoutedTermsToo(17 tests) holds case constant so only the anchor can refuse —OPMinDOPMA,BODinBODY,OMBinBOMBING, plus a collision at offset 0 where only the right-hand anchor can act. Three assertions per case so a negative can't pass for the wrong reason. No boundary bug was found;term_patternis correct. The test was the problem.The now-false docstring on the old class is corrected rather than left beside the class that disproves it.
Why boundaries matter, measured
Across 54,044 articles:
mailnistactSame probe, anchors stripped: an article about firing an FTC commissioner satisfies a mail-ballot probe via the word "email."
Design calls
MIN_PROBE_EVIDENCE = 2— a probe resting on one bare term is rejected at load.FedRAMPalone is matched by an AWS region-launch post.reinstat*), not inferred. The spec sketch wrote a bare stem; implicit prefix matching makes every term quietly broader than it reads, which is the direction that manufactures false confidence.INCONCLUSIVEis counted, never dropped. A probe set that decays to the ones still matching is a green light that means nothing.Verified
make checkexit 0, 1005 tests (988 → 1005). Read-only proven three ways:md5sum -con the corpus after every run, a CLI test whose sessioncommitraises, and a context test assertingsession.new/deleted/dirtyare empty. All new tests build synthetic corpora — none reads the live database or pins a real event.