Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
116 changes: 116 additions & 0 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -157,6 +157,53 @@ that decides what a brief looks like.

---

## The calibration loop: staking and grading your own calls

Every other ledger command reads. These four write, and they are what makes the graph
a calibration instrument rather than a reading list -- the thing being calibrated is
*your* judgement.

```bash
questions add "Does CMMC Phase 2 slip past its statutory date?" --cadence 90d
predict 23 "Yes -- slips" --by 2026-12-31 --confidence 0.7
forecast --due
resolve 41 --outcome no --note "DFARS class deviation published 2026-11-14"
```

All four are local database work: no `ANTHROPIC_API_KEY`, no network call. The loop is
`forecast --due` (what is due right now, at each question's own speed), grade what
resolved, read the diff on whatever came up for review, stake anything new. **The brief
is optional to that loop** -- ingestion is the fragile half, and the ledger should keep
working on a week where the sources returned nothing worth reading.

**A date and a confidence are required, and rejected at entry.** Nothing is stored
without them. The tool's first 33 predictions were graded zero times: 25 were phrased
"X would signal Y" -- an interpretation rule, which cannot be wrong -- and only 3
carried a date, so 19 aged out unjudged. A claim with no resolution date can never come
due. There is no confidence default either; an unstated confidence is a non-commitment
in a different costume.

**A cadence is not a deadline.** A question's `--cadence` says how often it is worth
re-examining; a prediction's `--by` says when a specific claim resolves. A question
reviewed quarterly can hold a claim resolving in three weeks. `forecast --due` tests
each question against its *own* interval -- a 7d question and a 90d question come due
independently -- and **stamps a question as reviewed whether or not anything moved**,
because a quiet question that reappears every day trains you to skim.

**`predictions track-record` counts your predictions only.** The model's are reported
under a separate heading and never folded into the figure; a hit rate that blends the
two measures nothing. Nothing auto-resolves from coverage: a tool that grades its
operator's calls using the corpus that produced them is measuring agreement with itself.

Schema: `questions.cadence`, `questions.last_reviewed_at`, and
`predictions.author | due_by | confidence | outcome`, added by
`python -m src.database.migrations.add_calibration_loop` (reversible; the downgrade
refuses rather than misattributing an operator claim). Commands live in
`src/cli/stake.py`, `src/cli/questions.py` and `src/cli/forecast.py`; interval
arithmetic in `src/utils/cadence.py`.

---

## Beats: briefing a subject instead of a person

The user profile models *you* -- a location, a profession, a voting context. A **beat**
Expand Down Expand Up @@ -271,6 +318,75 @@ reads as inactivity. A named individual may appear inside a rendered situation w
source document names a signatory -- an attribute of a document, which expires with it --
but never as a stored row, which would accumulate. See `docs/CONCEPTS.md`.

### Coverage probes: can the beat actually see its domain?

Added 2026-08-27. An article count tells you ingestion is *running*. It does not tell
you ingestion *reaches the domain*, and the two were conflated once already: a beat
resolving hundreds of Federal Register documents missed the reinstatement of the
FedRAMP director, because a personnel change is not a document. The corpus held three
incidental FedRAMP mentions across 50,983 articles and none within two weeks. Every
volume number was green.

A **coverage probe** is the test that catches it: name something that actually happened
in this domain, then check whether the beat can see it.

```json
"coverage_probes": [
{
"date": "2026-08-24",
"what": "FedRAMP director reinstated",
"terms": ["FedRAMP"],
"any_of": [["director", "administrator"], ["reinstat*", "return*", "restored"]]
}
]
```

```bash
insightweaver beat coverage us-public-sector-compliance
```

```
[MATCHED] OPM issued corrections to its final rules on reduction-in-force appeals
event 2026-08-25 | window 2026-08-11 .. 2026-09-08
4 matching article(s); earliest shown
seen by Federal Register - Documents API (2026-08-25)
Suitability Action Appeals; Correction

[UNMATCHED] FedRAMP director reinstated
no article from this beat's feeds matched
closest article lacked: FedRAMP
and no feed anywhere in the corpus carried it either.

2 matched + 1 unmatched + 1 inconclusive of 4 probe(s) declared
```

`terms` must all appear in one article; each `any_of` group needs one member. The two
levels exist because `FedRAMP` alone is too weak to be evidence -- an AWS region-launch
post mentions it -- while an exact headline is too brittle to survive a second outlet's
phrasing. A probe resting on one bare term is **rejected by the loader**, not accepted:
a probe that passes on generic terms manufactures the confidence this command exists to
remove.

Matching is deterministic word-boundary matching, no model call. That anchoring is what
makes a match evidence: of 54,044 articles stored on 2026-08-27, 657 titles contain the
substring `nist` -- "administration", "minister", "Afghanistan", "communist" -- and four
are about NIST. A trailing `*` marks a stem (`reinstat*` matches "reinstatement"); the
marker is explicit rather than inferred so the widening is visible to whoever has to
trust the result.

The command reports **which feed carried each match**, because "your expected source
carried it" and "an unrelated outlet mentioned it in passing" are different findings
with different repairs. An unmatched probe is re-checked across the whole corpus, so the
output distinguishes "nobody had it" from "feeds you do not subscribe to had it".

It exits non-zero so it can gate: `1` if any probe is unmatched, `2` if nothing could be
measured. A probe whose window predates the corpus is `INCONCLUSIVE` -- neither pass nor
fail -- and stays in the denominator, because a probe set that quietly decays to the
events still in retention is a green light that means nothing. That is also why an
all-inconclusive run and a beat with no probes both exit `2` rather than `0`.

Reads the article corpus only. No API key, no network, and it never writes.

Running your first beat needs the two beat tables, plus one each for institutional
activity and standing questions:

Expand Down
35 changes: 35 additions & 0 deletions config/beats/us-public-sector-compliance.json
Original file line number Diff line number Diff line change
Expand Up @@ -49,5 +49,40 @@
"Does any CISA BOD create a compliance obligation with a deadline inside 90 days?",
"Where do GovRAMP and TX-RAMP diverge from FedRAMP in ways that matter to a multi-state CSP?"
],
"coverage_probes": [
{
"date": "2026-08-25",
"what": "OPM issued corrections to its final rules on reduction-in-force, suitability and probationary appeals",
"terms": ["OPM"],
"any_of": [
["reduction in force", "suitability", "probationary"],
["final rule", "correction"]
]
},
{
"date": "2026-08-24",
"what": "Supreme Court let the executive order restricting mail-in voting take effect",
"terms": ["Supreme Court", "mail-in voting"],
"any_of": [["allow*", "restriction*", "sides with", "stay"]]
},
{
"date": "2026-08-24",
"what": "FedRAMP director reinstated",
"terms": ["FedRAMP"],
"any_of": [
["director", "administrator"],
["reinstat*", "return*", "restored"]
]
},
{
"date": "2021-11-03",
"what": "CISA issued Binding Operational Directive 22-01, creating the Known Exploited Vulnerabilities catalog and its remediation deadlines",
"terms": ["CISA"],
"any_of": [
["binding operational directive", "BOD 22-01"],
["known exploited", "catalog"]
]
}
],
"channels": ["terminal"]
}
24 changes: 17 additions & 7 deletions docs/CONCEPTS.md
Original file line number Diff line number Diff line change
Expand Up @@ -18,31 +18,41 @@ A sixth concept, the **Beat**, is not a commitment but a *scope* over them: it s

A **Question** is an unresolved epistemic thread the coverage is implicitly tracking. Most situations end with one — "Will the Fed cut rates in June?" — and that question persists across daily runs until it resolves.

A Question has one of two origins. Most are **emergent**: the graph notices what coverage leaves unresolved. A beat can also **declare** one up front — see "Standing questions" below. Both are the same kind of row, and everything downstream (predictions, decision evidence, appearance counts) treats them identically.
A Question has one of three origins. Most are **emergent**: the graph notices what coverage leaves unresolved. A beat can also **declare** one up front — see "Standing questions" below. Since 2026-08-27 the operator can declare one directly with `questions add "..." --cadence 90d`. All three are the same kind of row, and everything downstream (predictions, decision evidence, appearance counts) treats them identically.

**Cadence.** A question the operator declared carries a `cadence` — a review interval such as `7d`, `90d`, `1y`. **A cadence is not a deadline.** It says how often the question is worth re-examining; a Prediction's `due_by` says when a specific claim resolves. A question reviewed quarterly can hold a claim resolving in three weeks, and that nesting is the point: a CISA directive deadline is a 30-day loop and whether CMMC Phase 2 slips is a multi-year one, and forcing both onto one rhythm checks the fast one too late and buries the slow one in noise.

`forecast --due` surfaces a question when *its own* interval has elapsed since `last_reviewed_at` (or `first_asked_at`, if never reviewed), and **stamps it whether or not anything moved** — a quiet question that reappears every day trains the operator to skim. A cadence is itself falsifiable: a question always quiet at its interval was set too fast, one that moved twice before its next review was set too slow.

Cadence is nullable, and null means "not on a review schedule". Emergent questions have none — the interval is the operator's read on how fast a subject moves, so nothing infers it from coverage volume.

**When created.** Each situation's `unresolved_questions.primary` (and optional secondaries) is matched against the open question graph by a Haiku call. If today's question is the same underlying question as one already open, today's situation gets bound to that existing Question. Otherwise a new Question is created.

**When resolved.** Manually, via `insightweaver questions resolve <id> --note "..."`. The tool never auto-resolves; deciding a thread has closed is editorial.

**Reappearance.** When a previously resolved Question's text appears again in fresh coverage, a *new* Question is created with `previous_question_id` pointing at the resolved one. Resolution is never silently undone.

**CLI.** `questions list [--beat NAME] | show <id> | resolve <id> --note ...`
**CLI.** `questions add "..." --cadence 90d | list [--beat NAME] | show <id> | resolve <id> --note ...`. `questions list` shows each question's cadence and time until next review.

**In the brief.** Returning questions surface their identity inline: `Q47 (run 4, asked 2026-03-12)` for repeat appearances, `Q47 (new)` for first ones.

---

## Predictions

A **Prediction** is a falsifiable observable the synthesis committed to watching for. Each `what_to_watch` entry in a situation's output becomes a Prediction keyed to that situation's primary Question.
A **Prediction** is a falsifiable claim about the future. Every prediction carries an `author`: `model` or `operator`.

**Model predictions.** Each `what_to_watch` entry in a situation's output becomes a Prediction keyed to that situation's primary Question, created during synthesis after questions are resolved. Before each new synthesis, a check pass grades the open ledger against today's coverage: triggered, contradicted, expired (no signal after 90 days), or still open.

**Operator predictions.** Staked by hand: `predict <question-id> "..." --by YYYY-MM-DD --confidence 0.7`. Both flags are **required and rejected at entry** — nothing is stored without them. That is not fussiness. Of the first 33 model predictions, 25 were phrased "X would signal Y" (an interpretation rule, which cannot be wrong) and only 3 carried a date; 19 expired unjudged. A claim with no resolution date can never come due, so it can never be graded, and a confidence with a default is a non-commitment in a different costume.

**When created.** During synthesis, after questions are resolved. Each `{observable, trigger_condition}` pair in `where_this_goes.what_to_watch` becomes a Prediction row.
**When an operator prediction resolves.** Manually: `resolve <id> --outcome yes|no --note "..."`. `resolved_at` is recorded **separately from `due_by`** — resolving three months late is itself calibration data. A resolved prediction is not editable, and resolving one twice is refused.

**When resolved.** Before each new synthesis, a check pass grades the open ledger against today's coverage: triggered (observable appeared), contradicted (coverage explicitly went the other way), expired (no signal after 90 days), or still open.
**Auto-resolution from coverage is deliberately absent.** A tool that grades its operator's calls using the same corpus that produced them is measuring agreement with itself. The friction of resolving by hand is the instrument, not an obstacle to it.

**Why it matters.** This makes the tool's forward-looking statements auditable. The `predictions track-record` command shows the calibration: of N predictions resolved in the last 90 days, what fraction triggered vs. were contradicted.
**The author split.** `predictions track-record` reports the operator's plain hit rate — how often the claim, as written, came true — and reports model predictions under a separate heading. The two are never mixed: a track record blending them measures nothing. The model's stay in the ledger as *prompts*, suggestions about what is worth holding an opinion on.

**CLI.** `predictions open | triggered | contradicted | track-record`, each taking `--beat NAME`
**CLI.** `predictions open | triggered | contradicted | track-record`, each taking `--beat NAME`; plus the write side, `predict` and `resolve`.

**In the brief.** A transparency line at the top of each brief reports the check results: "Prediction check: 8 open observables graded — 1 triggered, 0 contradicted, 7 still open."

Expand Down
Loading
Loading