diff --git a/README.md b/README.md index 6b4384b..efb99ab 100644 --- a/README.md +++ b/README.md @@ -30,15 +30,21 @@ changes. Last updated 2026-08-27. **Known limitations, stated plainly** -- **The calibration record is empty.** 33 predictions have been generated and **zero** ever - graded; 25 of them are phrased "X would signal Y", which is an interpretation rule and cannot be - wrong. This is the tool's central claim and it is currently unmet. `backlog/011` is the fix, and - it moves prediction-making to the operator. -- **Coverage reaches documents, not events.** The first live compliance brief missed a major - personnel change in its own domain because no federal-IT trade outlet is configured. Across 207 - feeds there is not one. `backlog/009` and `backlog/010`. -- **Only one timescale.** Every question is reviewed on whatever rhythm the operator happens to - invoke, though the subjects move at very different speeds. Folded into `backlog/011`. +- **The calibration record is still empty, though the machinery now exists.** 33 model-generated + predictions have been graded zero times; 25 are phrased "X would signal Y", an interpretation + rule that cannot be wrong. As of `backlog/011` an operator can stake a dated, confidence-bearing + claim and resolve it, and those are counted separately from the model's. **Nothing has been + staked yet.** Until something is, the tool's central claim remains unmet -- the fix is a + mechanism, not a result. +- **Adding a feed to `config/feeds/` does not add it to the database.** `FeedManager.load_feeds_to_database()` + syncs config into the `rss_feeds` table, and the fetcher reads the table, not the config. A beat + resolves feeds from config, so a newly added source will resolve, pass its tests, and fetch + nothing until that sync is run. This was found on 2026-08-28 when five trade outlets were + configured, CI was green, and the corpus stayed empty of them. +- **Questions declared in a beat config carry no review cadence.** The four standing questions on + the `us-public-sector-compliance` beat -- including the CMMC one -- predate cadences, and there is + no `questions set-cadence`. They are invisible to `forecast --due` until re-declared with + `questions add --cadence`. Per-question cadences otherwise work (`backlog/011`). - **Per-run cost is unmeasured.** There is no token accounting. - **Delta-based features need history to mean anything.** Institutional activity reports "no trailing average yet" until several runs have accumulated. This is by construction, not a bug. diff --git a/SOURCES.md b/SOURCES.md index 5716379..d0de999 100644 --- a/SOURCES.md +++ b/SOURCES.md @@ -284,9 +284,11 @@ Stated plainly, because an unverified claim recorded as a fact is worse than no that section. **No quote in this file is unattributed, and none is unverified.** Also verified by fetching: that each of the five configured trade feeds returns a non-zero article count and stores rows, and that `content` is empty on the two GEMG feeds and populated on the three Scoop - News Group feeds. Verified by reading the stored row: that the FedScoop feed carries *"FedRAMP - Director Pete Waterman reinstated at GSA after weeks of leave"* with 320 words of body text — - the specific event the beat was blind to. + News Group feeds. Verified by reading the stored row: that the FedScoop feed carried the + FedRAMP leadership change of 2026-08-26 with 320 words of body text -- the specific event the + beat was blind to. The headline names the official; it is not reproduced here, because a durable + project artifact is not the "attribute of a specific document event" that + `backlog/006-institutional-activity.md` permits a person's name to appear as. - **Verified only as a negative:** that Scoop News Group publishes no terms page. Five candidate URLs were tried and all returned 404, and a link scan of the `fedscoop.com` homepage found no terms, privacy, legal or copyright link (claim 4). A page that exists somewhere those checks did diff --git a/backlog/009-federal-it-trade-press.md b/backlog/009-federal-it-trade-press.md index 40ba970..5e1cc73 100644 --- a/backlog/009-federal-it-trade-press.md +++ b/backlog/009-federal-it-trade-press.md @@ -1,6 +1,7 @@ # Add federal-IT trade press to the compliance beat, so it can see events and not only published documents. REPO: InsightWeaver -STATUS: QUEUED # QUEUED | IN_PROGRESS | PARKED | DONE | FAILED +STATUS: DONE # QUEUED | IN_PROGRESS | PARKED | DONE | FAILED +LANDED: PR #16, merged 2026-08-28 ACCEPTANCE: `make check` passes, plus: at least four federal-IT trade outlets are configured in `config/feeds/` with the beat's existing tags (`regulatory`, `federal_policy`, `cybersecurity`); each has a row in `SOURCES.md` recording its basis for use; the `us-public-sector-compliance` beat resolves them; a live fetch retrieves a non-zero article count from each, and that count is reported in the PR; and `tests/config/test_beats.py` asserts the beat's resolved set now contains at least one non-`.gov` trade source, so a future feed-config edit that silently drops them fails CI. OUT OF SCOPE: Any source whose terms forbid automated retrieval. Commercial wire services (AP, Reuters) -- rule 2 in `SOURCES.md` stands unchanged and is separate from this task. HTML scraping adapters; every outlet in scope publishes RSS, and if one does not, drop it rather than writing a scraper. Paywalled content. Changing clustering, synthesis, entity matching, or standing questions. Re-tuning the Federal Register filter. Backfilling history. LANDMINES: **The failure this task fixes was invisible for exactly the reason it will be tempting to declare this done early.** Adding feeds makes article counts go up, and a rising count is not evidence the beat can see an event -- that was the mistake in task 005, which measured 469 documents narrowed to 24 and called the domain covered. Verify against task 010's known-event test, not against volume. Trade RSS feeds are often truncated to a summary with the body behind a click: check that `content` is populated and not just `description`, because a title-only corpus will match entity aliases while giving synthesis nothing to analyze. Several federal-IT outlets publish very high volume including vendor press releases and sponsored posts, which will drown the beat's 50-article curation window if unfiltered -- prefer a topic or category feed over a firehose where the outlet offers one. `main` is protected (5 required checks, `enforce_admins: true`) -- open a PR, do not push to it. @@ -69,9 +70,13 @@ beat, all returning a non-zero live count. `tests/config/test_beats.py` now asse resolves at least one non-`.gov` source and names all five, and the assertion was checked by removing the config file and watching it fail. -**The evidence that matters is not the count.** FedScoop's feed carried -*"FedRAMP Director Pete Waterman reinstated at GSA after weeks of leave"*, and it stored with 320 -words of body text. That is the exact event this task exists because the beat missed. +**The evidence that matters is not the count.** FedScoop's feed carried the FedRAMP leadership +change of 2026-08-26, and it stored with 320 words of body text. That is the exact event this task +exists because the beat missed. + +The headline names the official. It is not reproduced here: task 006 permits a person's name to +appear as an attribute of a specific document event, and a task file is not that -- it is a durable +artifact in a public repository, which is the accumulation that boundary exists to prevent. **Two decisions a reviewer must make rather than accept:** diff --git a/backlog/010-known-event-coverage-test.md b/backlog/010-known-event-coverage-test.md index 292641f..065db37 100644 --- a/backlog/010-known-event-coverage-test.md +++ b/backlog/010-known-event-coverage-test.md @@ -1,6 +1,7 @@ # Test a beat against events that actually happened, so "can it see the domain" stops being answered with article counts. REPO: InsightWeaver -STATUS: QUEUED # QUEUED | IN_PROGRESS | PARKED | DONE | FAILED +STATUS: DONE # QUEUED | IN_PROGRESS | PARKED | DONE | FAILED +LANDED: PR #17, merged 2026-08-28 ACCEPTANCE: `make check` passes, plus: a beat may declare a `coverage_probes` list of known past events, each with a date, a short description, and the distinctive terms that would appear in any report of it; a new `insightweaver beat coverage NAME` command reports, per probe, whether the corpus contains a matching article and which feed carried it; the command exits **non-zero when any probe is unmatched**, so it can gate; matching is deterministic word-boundary term matching with no LLM call; and a probe whose window predates the corpus is reported as `INCONCLUSIVE` rather than counted as either a pass or a failure. OUT OF SCOPE: Running the pipeline, synthesis, or any Anthropic call -- this command reads the article corpus only and must work with no API key. Judging whether a *brief* covered an event well; this measures ingestion reach, not analysis quality. Auto-generating probes. Fetching anything. Changing feeds, adapters, clustering, or the standing agenda. Backfilling history to make an old probe match. LANDMINES: **A probe that passes because its terms are generic is worse than no probe**, since it manufactures the exact confidence this task exists to remove -- require terms distinctive enough that a match is evidence, and test the collision cases the way task 006 did (of 50,983 stored articles, 617 titles contain the substring "nist" and 2 are about NIST). The corpus is shared and read-only: this command must never write. A probe list will rot as the corpus ages past its retention -- that is what `INCONCLUSIVE` is for, and an inconclusive probe must not be silently dropped from the count, because a probe set that quietly shrinks to the ones that still match is a green light that means nothing. Do not make this a pytest fixture with hardcoded events; it is an operator command about live corpus state, and pinning it into CI would make the suite fail for reasons unrelated to the code under test. `main` is protected (5 required checks, `enforce_admins: true`) -- open a PR, do not push to it. diff --git a/backlog/011-user-calibration-loop.md b/backlog/011-user-calibration-loop.md index ee57ccc..63cde70 100644 --- a/backlog/011-user-calibration-loop.md +++ b/backlog/011-user-calibration-loop.md @@ -1,6 +1,7 @@ # Let the operator stake and resolve their own falsifiable predictions on questions that each run at their own cadence, so the track record measures the person and the loops run at the speed the subject actually moves. REPO: InsightWeaver -STATUS: QUEUED # QUEUED | IN_PROGRESS | PARKED | DONE | FAILED +STATUS: DONE # QUEUED | IN_PROGRESS | PARKED | DONE | FAILED +LANDED: PR #15, merged 2026-08-28 ACCEPTANCE: `make check` passes, plus: four new write commands -- `questions add`, `predict`, `resolve`, and `forecast --due` -- let a human declare a question, stake a claim against it with a **required** resolution date and a confidence, see what has come due, and record an outcome with a note; a prediction with no date is **rejected at entry**, not stored; predictions carry an author (`operator` | `model`) and `predictions track-record` counts **operator predictions only**, with model predictions reported separately and never mixed into the calibration figure; **each question carries a `cadence`** (a review interval, e.g. `7d`, `90d`, `1y`) and `forecast --due` surfaces a question when its own interval has elapsed since it was last reviewed -- not on a single global rhythm -- with `questions list` showing each question's cadence and time-until-next-review; reviewing a question stamps it whether or not anything moved, so a quiet question does not reappear daily; every command works with no `ANTHROPIC_API_KEY` and makes no network call; and tests cover the date requirement, the author split in track-record, resolving a prediction that is not yet due, resolving one twice, a question whose cadence has not elapsed being absent from `--due`, and two questions on different cadences coming due independently. OUT OF SCOPE: Inferring or auto-tuning a question's cadence from coverage volume -- the operator sets it, because the interval is a claim about how fast the subject moves and that is the operator's read to make. Any scheduler, daemon, cron, or background process; `--due` is a question you ask, not a thing that runs. Changing how the model generates predictions, or the synthesis prompts -- the model's predictions stay exactly as they are and simply stop counting toward calibration. Scoring, weighting, or Brier-style aggregate math beyond a plain hit rate; that is a later decision once there is real data. Auto-resolving a prediction from coverage -- the human resolves, deliberately, because a tool that grades its own operator's calls is not a calibration instrument. Reminders, notifications, email, or scheduling. Editing or deleting a resolved prediction. Any change to beats, sources, adapters, entity matching, or the standing agenda. LANDMINES: **The reason the ledger is empty today is not that nobody ran a resolver.** Of 33 model-generated predictions, 25 are phrased "X would signal Y" -- interpretation rules, which cannot be wrong -- and only 3 contain a date. 19 expired ungraded because there was nothing gradeable in them. If `predict` accepts a claim with no date or no observable outcome, this task rebuilds that failure with a nicer interface, which is why the date is a hard reject rather than a default. Do not add a confidence *default*: a stake with an unstated confidence is the same non-commitment in a different costume. `resolve` must record the resolution date separately from the due date -- resolving three months late is itself calibration data. **A cadence is not a deadline**: it says how often the question is worth re-examining, while a prediction's `--by` date says when a specific claim resolves. Conflating them collapses the nesting this task exists to build -- a question reviewed quarterly can hold a claim that resolves in three weeks. The existing `predictions` table has `observable_text`, `trigger_condition`, `status`, `resolved_at`, `resolution_note`: extend it, do not build a parallel table. `main` is protected (5 required checks, `enforce_admins: true`) -- open a PR, do not push to it.