Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
24 changes: 15 additions & 9 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -30,15 +30,21 @@ changes. Last updated 2026-08-27.

**Known limitations, stated plainly**

- **The calibration record is empty.** 33 predictions have been generated and **zero** ever
graded; 25 of them are phrased "X would signal Y", which is an interpretation rule and cannot be
wrong. This is the tool's central claim and it is currently unmet. `backlog/011` is the fix, and
it moves prediction-making to the operator.
- **Coverage reaches documents, not events.** The first live compliance brief missed a major
personnel change in its own domain because no federal-IT trade outlet is configured. Across 207
feeds there is not one. `backlog/009` and `backlog/010`.
- **Only one timescale.** Every question is reviewed on whatever rhythm the operator happens to
invoke, though the subjects move at very different speeds. Folded into `backlog/011`.
- **The calibration record is still empty, though the machinery now exists.** 33 model-generated
predictions have been graded zero times; 25 are phrased "X would signal Y", an interpretation
rule that cannot be wrong. As of `backlog/011` an operator can stake a dated, confidence-bearing
claim and resolve it, and those are counted separately from the model's. **Nothing has been
staked yet.** Until something is, the tool's central claim remains unmet -- the fix is a
mechanism, not a result.
- **Adding a feed to `config/feeds/` does not add it to the database.** `FeedManager.load_feeds_to_database()`
syncs config into the `rss_feeds` table, and the fetcher reads the table, not the config. A beat
resolves feeds from config, so a newly added source will resolve, pass its tests, and fetch
nothing until that sync is run. This was found on 2026-08-28 when five trade outlets were
configured, CI was green, and the corpus stayed empty of them.
- **Questions declared in a beat config carry no review cadence.** The four standing questions on
the `us-public-sector-compliance` beat -- including the CMMC one -- predate cadences, and there is
no `questions set-cadence`. They are invisible to `forecast --due` until re-declared with
`questions add --cadence`. Per-question cadences otherwise work (`backlog/011`).
- **Per-run cost is unmeasured.** There is no token accounting.
- **Delta-based features need history to mean anything.** Institutional activity reports "no
trailing average yet" until several runs have accumulated. This is by construction, not a bug.
Expand Down
8 changes: 5 additions & 3 deletions SOURCES.md
Original file line number Diff line number Diff line change
Expand Up @@ -284,9 +284,11 @@ Stated plainly, because an unverified claim recorded as a fact is worse than no
that section. **No quote in this file is unattributed, and none is unverified.** Also verified by
fetching: that each of the five configured trade feeds returns a non-zero article count and
stores rows, and that `content` is empty on the two GEMG feeds and populated on the three Scoop
News Group feeds. Verified by reading the stored row: that the FedScoop feed carries *"FedRAMP
Director Pete Waterman reinstated at GSA after weeks of leave"* with 320 words of body text —
the specific event the beat was blind to.
News Group feeds. Verified by reading the stored row: that the FedScoop feed carried the
FedRAMP leadership change of 2026-08-26 with 320 words of body text -- the specific event the
beat was blind to. The headline names the official; it is not reproduced here, because a durable
project artifact is not the "attribute of a specific document event" that
`backlog/006-institutional-activity.md` permits a person's name to appear as.
- **Verified only as a negative:** that Scoop News Group publishes no terms page. Five candidate
URLs were tried and all returned 404, and a link scan of the `fedscoop.com` homepage found no
terms, privacy, legal or copyright link (claim 4). A page that exists somewhere those checks did
Expand Down
13 changes: 9 additions & 4 deletions backlog/009-federal-it-trade-press.md
Original file line number Diff line number Diff line change
@@ -1,6 +1,7 @@
# Add federal-IT trade press to the compliance beat, so it can see events and not only published documents.
REPO: InsightWeaver
STATUS: QUEUED # QUEUED | IN_PROGRESS | PARKED | DONE | FAILED
STATUS: DONE # QUEUED | IN_PROGRESS | PARKED | DONE | FAILED
LANDED: PR #16, merged 2026-08-28
ACCEPTANCE: `make check` passes, plus: at least four federal-IT trade outlets are configured in `config/feeds/` with the beat's existing tags (`regulatory`, `federal_policy`, `cybersecurity`); each has a row in `SOURCES.md` recording its basis for use; the `us-public-sector-compliance` beat resolves them; a live fetch retrieves a non-zero article count from each, and that count is reported in the PR; and `tests/config/test_beats.py` asserts the beat's resolved set now contains at least one non-`.gov` trade source, so a future feed-config edit that silently drops them fails CI.
OUT OF SCOPE: Any source whose terms forbid automated retrieval. Commercial wire services (AP, Reuters) -- rule 2 in `SOURCES.md` stands unchanged and is separate from this task. HTML scraping adapters; every outlet in scope publishes RSS, and if one does not, drop it rather than writing a scraper. Paywalled content. Changing clustering, synthesis, entity matching, or standing questions. Re-tuning the Federal Register filter. Backfilling history.
LANDMINES: **The failure this task fixes was invisible for exactly the reason it will be tempting to declare this done early.** Adding feeds makes article counts go up, and a rising count is not evidence the beat can see an event -- that was the mistake in task 005, which measured 469 documents narrowed to 24 and called the domain covered. Verify against task 010's known-event test, not against volume. Trade RSS feeds are often truncated to a summary with the body behind a click: check that `content` is populated and not just `description`, because a title-only corpus will match entity aliases while giving synthesis nothing to analyze. Several federal-IT outlets publish very high volume including vendor press releases and sponsored posts, which will drown the beat's 50-article curation window if unfiltered -- prefer a topic or category feed over a firehose where the outlet offers one. `main` is protected (5 required checks, `enforce_admins: true`) -- open a PR, do not push to it.
Expand Down Expand Up @@ -69,9 +70,13 @@ beat, all returning a non-zero live count. `tests/config/test_beats.py` now asse
resolves at least one non-`.gov` source and names all five, and the assertion was checked by
removing the config file and watching it fail.

**The evidence that matters is not the count.** FedScoop's feed carried
*"FedRAMP Director Pete Waterman reinstated at GSA after weeks of leave"*, and it stored with 320
words of body text. That is the exact event this task exists because the beat missed.
**The evidence that matters is not the count.** FedScoop's feed carried the FedRAMP leadership
change of 2026-08-26, and it stored with 320 words of body text. That is the exact event this task
exists because the beat missed.

The headline names the official. It is not reproduced here: task 006 permits a person's name to
appear as an attribute of a specific document event, and a task file is not that -- it is a durable
artifact in a public repository, which is the accumulation that boundary exists to prevent.

**Two decisions a reviewer must make rather than accept:**

Expand Down
3 changes: 2 additions & 1 deletion backlog/010-known-event-coverage-test.md
Original file line number Diff line number Diff line change
@@ -1,6 +1,7 @@
# Test a beat against events that actually happened, so "can it see the domain" stops being answered with article counts.
REPO: InsightWeaver
STATUS: QUEUED # QUEUED | IN_PROGRESS | PARKED | DONE | FAILED
STATUS: DONE # QUEUED | IN_PROGRESS | PARKED | DONE | FAILED
LANDED: PR #17, merged 2026-08-28
ACCEPTANCE: `make check` passes, plus: a beat may declare a `coverage_probes` list of known past events, each with a date, a short description, and the distinctive terms that would appear in any report of it; a new `insightweaver beat coverage NAME` command reports, per probe, whether the corpus contains a matching article and which feed carried it; the command exits **non-zero when any probe is unmatched**, so it can gate; matching is deterministic word-boundary term matching with no LLM call; and a probe whose window predates the corpus is reported as `INCONCLUSIVE` rather than counted as either a pass or a failure.
OUT OF SCOPE: Running the pipeline, synthesis, or any Anthropic call -- this command reads the article corpus only and must work with no API key. Judging whether a *brief* covered an event well; this measures ingestion reach, not analysis quality. Auto-generating probes. Fetching anything. Changing feeds, adapters, clustering, or the standing agenda. Backfilling history to make an old probe match.
LANDMINES: **A probe that passes because its terms are generic is worse than no probe**, since it manufactures the exact confidence this task exists to remove -- require terms distinctive enough that a match is evidence, and test the collision cases the way task 006 did (of 50,983 stored articles, 617 titles contain the substring "nist" and 2 are about NIST). The corpus is shared and read-only: this command must never write. A probe list will rot as the corpus ages past its retention -- that is what `INCONCLUSIVE` is for, and an inconclusive probe must not be silently dropped from the count, because a probe set that quietly shrinks to the ones that still match is a green light that means nothing. Do not make this a pytest fixture with hardcoded events; it is an operator command about live corpus state, and pinning it into CI would make the suite fail for reasons unrelated to the code under test. `main` is protected (5 required checks, `enforce_admins: true`) -- open a PR, do not push to it.
Expand Down
3 changes: 2 additions & 1 deletion backlog/011-user-calibration-loop.md
Original file line number Diff line number Diff line change
@@ -1,6 +1,7 @@
# Let the operator stake and resolve their own falsifiable predictions on questions that each run at their own cadence, so the track record measures the person and the loops run at the speed the subject actually moves.
REPO: InsightWeaver
STATUS: QUEUED # QUEUED | IN_PROGRESS | PARKED | DONE | FAILED
STATUS: DONE # QUEUED | IN_PROGRESS | PARKED | DONE | FAILED
LANDED: PR #15, merged 2026-08-28
ACCEPTANCE: `make check` passes, plus: four new write commands -- `questions add`, `predict`, `resolve`, and `forecast --due` -- let a human declare a question, stake a claim against it with a **required** resolution date and a confidence, see what has come due, and record an outcome with a note; a prediction with no date is **rejected at entry**, not stored; predictions carry an author (`operator` | `model`) and `predictions track-record` counts **operator predictions only**, with model predictions reported separately and never mixed into the calibration figure; **each question carries a `cadence`** (a review interval, e.g. `7d`, `90d`, `1y`) and `forecast --due` surfaces a question when its own interval has elapsed since it was last reviewed -- not on a single global rhythm -- with `questions list` showing each question's cadence and time-until-next-review; reviewing a question stamps it whether or not anything moved, so a quiet question does not reappear daily; every command works with no `ANTHROPIC_API_KEY` and makes no network call; and tests cover the date requirement, the author split in track-record, resolving a prediction that is not yet due, resolving one twice, a question whose cadence has not elapsed being absent from `--due`, and two questions on different cadences coming due independently.
OUT OF SCOPE: Inferring or auto-tuning a question's cadence from coverage volume -- the operator sets it, because the interval is a claim about how fast the subject moves and that is the operator's read to make. Any scheduler, daemon, cron, or background process; `--due` is a question you ask, not a thing that runs. Changing how the model generates predictions, or the synthesis prompts -- the model's predictions stay exactly as they are and simply stop counting toward calibration. Scoring, weighting, or Brier-style aggregate math beyond a plain hit rate; that is a later decision once there is real data. Auto-resolving a prediction from coverage -- the human resolves, deliberately, because a tool that grades its own operator's calls is not a calibration instrument. Reminders, notifications, email, or scheduling. Editing or deleting a resolved prediction. Any change to beats, sources, adapters, entity matching, or the standing agenda.
LANDMINES: **The reason the ledger is empty today is not that nobody ran a resolver.** Of 33 model-generated predictions, 25 are phrased "X would signal Y" -- interpretation rules, which cannot be wrong -- and only 3 contain a date. 19 expired ungraded because there was nothing gradeable in them. If `predict` accepts a claim with no date or no observable outcome, this task rebuilds that failure with a nicer interface, which is why the date is a hard reject rather than a default. Do not add a confidence *default*: a stake with an unstated confidence is the same non-commitment in a different costume. `resolve` must record the resolution date separately from the due date -- resolving three months late is itself calibration data. **A cadence is not a deadline**: it says how often the question is worth re-examining, while a prediction's `--by` date says when a specific claim resolves. Conflating them collapses the nesting this task exists to build -- a question reviewed quarterly can hold a claim that resolves in three weeks. The existing `predictions` table has `observable_text`, `trigger_condition`, `status`, `resolved_at`, `resolution_note`: extend it, do not build a parallel table. `main` is protected (5 required checks, `enforce_admins: true`) -- open a PR, do not push to it.
Expand Down
Loading