Skip to content

beats: test coverage against events that actually happened - #17

Merged
sam-aydlette merged 4 commits into
mainfrom
task/010-coverage-test
Aug 28, 2026
Merged

sam-aydlette merged 4 commits into
mainfrom
task/010-coverage-test

Conversation

@sam-aydlette

@sam-aydlette sam-aydlette commented Aug 27, 2026 •

Copy link
Copy Markdown
Owner

Implements backlog/010-known-event-coverage-test.md.

Merge #15 (calibration loop) first. This branch is based on task/011-calibration — both add a CLI command, and basing avoided a registration collision. Its diff carries 011's commit; once #15 lands this rebases clean.

Why

Task 005 asked "can the beat reach its domain" and answered with document counts — 469 Federal Register documents narrowed to 24, 38 in seven days. Both real. Both green. Neither could detect that the beat was structurally unable to see a personnel change, because those are not documents. The first live brief then missed the reinstatement of the FedRAMP director.

Article count measures whether ingestion is running, not whether it reaches the domain. This is the test that replaces it:

Name three things that actually happened in this domain this month, then check whether the beat can see them.

insightweaver beat coverage NAME

A beat declares coverage_probes — known past events plus the distinctive terms any report would carry. The command reports per probe, names which feed carried it, and exits non-zero when any probe is unmatched, so it can gate.

[MATCHED] OPM issued corrections to its final rules on RIF, suitability and probationary appeals
  requires OPM AND (reduction in force OR suitability OR probationary) AND (final rule OR correction)
  4 matching article(s) | seen by Federal Register - Documents API (2026-08-25)

[UNMATCHED] FedRAMP director reinstated
  54 article(s) from this beat's feeds in window
    no article from this beat's feeds matched -- closest article lacked: FedRAMP
    but 1 article(s) elsewhere in the corpus matched, outside this beat's sources:
      FedScoop (2026-08-26)
        FedRAMP Director reinstated at GSA after weeks of leave  [name redacted]
      the sources are reachable; this beat's list is what missed.

[INCONCLUSIVE] CISA issued Binding Operational Directive 22-01
  window ends 2021-11-17, before the beat's oldest stored article -- the event predates the corpus
    neither a pass nor a failure; still counted below.

2 matched + 1 unmatched + 1 inconclusive of 4 probe(s)
FAIL: 1 event(s) this beat claims to cover left no trace in its sources.  [exit 1]

That elsewhere line is the point: it separates "nobody carried it" from "feeds you don't subscribe to carried it" — different problems with different fixes. Run against a corpus containing task 009's sources, it diagnoses the state exactly.

The reviewer found the test guarding this was inert

TestSubstringMatchingWouldLie carried the task's central claim — that word-boundary matching is load-bearing. Its cases are confounded by acronym case-sensitivity: every collision is lowercase and every term is shouted, so the case rule rejects them before the anchors are ever consulted. And the shipped beat's own probes use exactly those terms — OPM, CISA.

I reproduced it by deleting LEFT_BOUNDARY and RIGHT_BOUNDARY from production code:

anchors stripped
old class 12 passed — notices nothing
new class 13 failed, 4 passed (positive controls)
restored 1005 passed

TestBoundariesAreLoadBearingForShoutedTermsToo (17 tests) holds case constant so only the anchor can refuse — OPM in DOPMA, BOD in BODY, OMB in BOMBING, plus a collision at offset 0 where only the right-hand anchor can act. Three assertions per case so a negative can't pass for the wrong reason. No boundary bug was found; term_pattern is correct. The test was the problem.

The now-false docstring on the old class is corrected rather than left beside the class that disproves it.

Why boundaries matter, measured

Across 54,044 articles:

term substring word-boundary false
mail 1,842 339 1,503
nist 5,364 73 5,291
act 13,254 1,933 11,321

Same probe, anchors stripped: an article about firing an FTC commissioner satisfies a mail-ballot probe via the word "email."

Design calls

  • MIN_PROBE_EVIDENCE = 2 — a probe resting on one bare term is rejected at load. FedRAMP alone is matched by an AWS region-launch post.
  • Stems are explicit (reinstat*), not inferred. The spec sketch wrote a bare stem; implicit prefix matching makes every term quietly broader than it reads, which is the direction that manufactures false confidence.
  • Exit 2 for "nothing measured" (no probes, or all inconclusive) beyond the written acceptance — an all-inconclusive run exiting 0 is exactly the green light the landmine warns about.
  • INCONCLUSIVE is counted, never dropped. A probe set that decays to the ones still matching is a green light that means nothing.

Verified

make check exit 0, 1005 tests (988 → 1005). Read-only proven three ways: md5sum -c on the corpus after every run, a CLI test whose session commit raises, and a context test asserting session.new/deleted/dirty are empty. All new tests build synthetic corpora — none reads the live database or pins a real event.

…orecast --due

The ledger held 33 predictions and had graded none. 25 were phrased "X would
signal Y" -- interpretation rules, which cannot be wrong -- and only 3 carried a
date, so 19 expired unjudged. The ledger was not empty for want of a resolver;
it was empty because nothing gradeable had been written into it, and because
every command was read-only: the model wrote the graph and the human read it.

Four write commands, all local (no API key, no network call):

  questions add "..." --cadence 90d
  predict 23 "Yes -- slips" --by 2026-12-31 --confidence 0.7
  forecast --due
  resolve 41 --outcome no --note "..."

--by and --confidence are required and rejected at entry, not defaulted. A claim
with no resolution date can never come due, so it can never be graded; a
confidence with a default is a non-commitment in a different costume. Both
refusals store nothing.

A cadence is not a deadline. questions.cadence says how often a question is
worth re-examining; predictions.due_by says when a specific claim resolves, so a
question reviewed quarterly can hold a claim resolving in three weeks.
forecast --due tests each question against its own interval, and stamps
last_reviewed_at whether or not anything moved -- a quiet question that
reappears daily trains the operator to skim.

predictions.author (operator | model) splits track-record: a plain hit rate for
the operator, model predictions under their own heading, never mixed. Nothing
auto-resolves from coverage; the human resolves, and resolved_at is recorded
separately from due_by so grading three months late stays visible. A resolved
prediction is not editable.

Schema changes are in a reversible migration, add_calibration_loop. The
predictions half is a SQLite table rebuild (relaxing made_in_synthesis_id to
nullable, which ALTER TABLE cannot do) with row counts asserted on both sides;
verified carrying all 33 existing rows, backfilled to author='model'. The
downgrade refuses rather than misattributing an operator claim.

track-record's output changed shape, so its two existing assertions were updated
to read the model heading. Interactive dispatch now splits arguments with shlex,
without which a multi-word claim reached Click as four arguments.
… counts

Task 005 asked whether a beat could reach its domain and answered with
document counts -- 469 Federal Register documents narrowed to 24, 38 in
seven days. Both real, both green, and neither able to detect that the
beat was structurally unable to see a personnel change, because a
personnel change is not a document. The first live brief then missed the
reinstatement of the FedRAMP director. The corpus held 3 incidental
FedRAMP mentions across 50,983 articles and none within two weeks.

An article count measures whether ingestion is running. It does not
measure whether ingestion reaches the domain. `insightweaver beat
coverage NAME` measures the second thing: a beat declares known past
events as `coverage_probes`, and the command reports per probe whether a
feed the beat subscribes to carried a matching article, and which one.

- `terms` must all appear; each `any_of` group needs one member. A probe
  resting on a single bare term is rejected by the loader -- `FedRAMP`
  alone is matched by an AWS region-launch post, and a probe that passes
  on generic terms manufactures the confidence this exists to remove.
- Matching is deterministic word-boundary matching through the same
  primitives coverage entities use. No model call. Measured on the
  corpus: 1,842 articles contain the substring "mail", 339 contain it as
  a word; 5,364 contain "nist" against 73 as a word.
- A trailing `*` marks a stem (`reinstat*`). Explicit rather than
  inferred, so the widening is visible to whoever has to trust the result.
- A probe whose window holds no articles from the beat's sources is
  INCONCLUSIVE, with the reason distinguishing "predates the corpus"
  from "corpus has not reached it" from a gap. It stays in the
  denominator; a probe set that decays to the events still in retention
  is a green light that means nothing.
- An unmatched probe is re-checked corpus-wide, so "nobody carried it"
  and "feeds you do not subscribe to carried it" are distinguishable.
- Exits 1 on any unmatched probe, 2 when nothing was measured (no probes
  declared, or all inconclusive). Reads only; never writes.

The shipped beat declares four probes. Against the current corpus two
match, the FedRAMP reinstatement does not, and BOD 22-01 predates the
corpus -- which is the honest measurement task 010 was written to get.
TestSubstringMatchingWouldLie carried this task's central claim -- that
word-boundary matching is load-bearing -- but could not actually check
it. Every collision in NIST_COLLISIONS is lowercase, and a shouted term
like NIST compiles case-sensitively, so case-sensitivity rejected those
headlines before the boundary anchors were ever consulted. Measured by
stripping the anchors from term_pattern in place, leaving the case rule
untouched: that class stays fully green, 12 passed.

That is not academic. The shipped beat's probes use shouted terms -- OPM
and CISA -- so a regression that dropped the anchoring would have left
exactly the production term type unguarded.

The new class holds case constant instead: term and headline are both
all-caps, so the boundary anchor is the only thing left that can reject
the match. OPM inside DOPMA and TOPMOST, BOD inside BODY, OMB inside
BOMBING, CISA inside PRECISA. Each case pairs the negative assertion
with a check that the probe's *other* required term is present, so a
headline that merely lacked everything could not satisfy it, and with a
positive control asserting the same probe still matches the term
standing alone.

Verified by the same method: with the anchors stripped from the
production regex, 13 of the 17 fail and the 4 that pass are the positive
controls, which should still match.

No change to the matcher, the schema, the beat config, the CLI, or any
existing test.
TestSubstringMatchingWouldLie said 'The boundary, not the two-level shape, is
what rejects these.' That is false for its own cases: every term in it is
shouted, so the compiled pattern is case-sensitive and the lowercase
collisions are refused before the anchors are consulted. Deleting
LEFT_BOUNDARY and RIGHT_BOUNDARY from entity_matcher leaves all 12 passing.

The IC that found this correctly declined to edit an existing test outside
its scope and flagged it instead. Correcting it here rather than leaving a
demonstrably false claim next to the class that disproves it.
@sam-aydlette
sam-aydlette merged commit f5479cf into main Aug 28, 2026
5 checks passed
@sam-aydlette
sam-aydlette deleted the task/010-coverage-test branch August 28, 2026 14:19
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant