From fc6428caa0812b161fc324109cdbee95fb5677ab Mon Sep 17 00:00:00 2001 From: sam-aydlette Date: Mon, 31 Aug 2026 11:49:18 -0400 Subject: [PATCH 1/3] position/watch: the atomic units, with so_what enforced in the schema Backlog task 013. Position holds the operator's decisions; a Watch is one pre-registered claim serving one of them. Neither file is in this repo -- they name real decisions, deadlines and exposures, and this repo is public. The loaders take configurable paths (POSITION_PATH / WATCHES_PATH) defaulting to ~/.config/insightweaver/, .gitignore refuses a real position.yaml, and what ships here is the schema plus config/{position,watches}.example.yaml. Three shape decisions, recorded rather than discovered: Watch is a new table, not an evolution of Decision + Factor. Task 012 dropped decision_factors and decisions at 0 rows, so there was nothing left to evolve. Decision lives in Position, not the database: it is an operator commitment in a hand-edited file under git, not a row the system writes. One model of the idea, not two. so_what is a key plus prose. `decision` is a key into the Position's decisions and `because` says what changes. The key makes invariant 2 machine-checkable; an invariant enforced only by prose is what left 25 unfalsifiable predictions in the deleted ledger. triggers is structured and must stay so: a list of clauses over terms, entities and source allowlists -- a disjunction of conjunctions Tier 1 can compile to word-boundary regexes with no model involved. predictions. trigger_condition was free text; 33 rows were written against it and graded zero times, because nothing downstream can evaluate a sentence. A clause that is a sentence is rejected by name. Rejection is a claim about the table, so it is tested against the table: every invalid watch asserts the row count is unchanged, and the assertion runs whether or not the loader raised. Empty, whitespace-only and absent so_what, an unknown decision, belief outside [0,1], a past expiry and staleness under one day all refuse and store nothing; a file with one bad watch stores none of its watches. Four CHECK constraints enforce the same invariants against raw SQL, so a row inserted by hand fails the way a bad file does. There is no `watch add` and there will not be one. It satisfies no acceptance criterion and is the seam through which auto-generated watches arrive (invariant 6). `watch sync` mirrors the files and takes no arguments; a test asserts on the registered subcommands and their parameter lists. staleness_alert_days accepts a cadence string via src/utils/cadence.py rather than a second interval parser. A Position over two pages warns loudly and still loads -- a three-page Position of real deadlines beats a one-page Position of vague interests, so length can only ever be a flag. Missing stakes and passed deadlines warn the same way. Everything structurally checkable rejects. 554 tests pass, 103 of them new. Every enforcement above was verified by mutation: removing it makes a named test fail. Co-Authored-By: Claude Opus 5 (1M context) Claude-Session: https://claude.ai/code/session_01M9gY2miAUXijbFm25YnFcq --- .env.example | 5 + .gitignore | 11 + Makefile | 9 +- backlog/022-open-questions.md | 22 +- config/position.example.yaml | 52 ++ config/watches.example.yaml | 94 +++ src/cli/app.py | 22 +- src/cli/watch.py | 146 +++++ src/config/settings.py | 14 + src/database/migrations/add_watches_table.py | 89 +++ src/database/models.py | 66 ++- src/position/__init__.py | 42 ++ src/position/position.py | 318 ++++++++++ src/position/watches.py | 583 +++++++++++++++++++ tests/cli/test_app.py | 19 +- tests/cli/test_watch_cli.py | 178 ++++++ tests/database/test_add_watches_table.py | 89 +++ tests/database/test_connection.py | 9 +- tests/database/test_models.py | 7 +- tests/position/__init__.py | 0 tests/position/conftest.py | 118 ++++ tests/position/test_position.py | 196 +++++++ tests/position/test_watches.py | 337 +++++++++++ 23 files changed, 2412 insertions(+), 14 deletions(-) create mode 100644 config/position.example.yaml create mode 100644 config/watches.example.yaml create mode 100644 src/cli/watch.py create mode 100644 src/database/migrations/add_watches_table.py create mode 100644 src/position/__init__.py create mode 100644 src/position/position.py create mode 100644 src/position/watches.py create mode 100644 tests/cli/test_watch_cli.py create mode 100644 tests/database/test_add_watches_table.py create mode 100644 tests/position/__init__.py create mode 100644 tests/position/conftest.py create mode 100644 tests/position/test_position.py create mode 100644 tests/position/test_watches.py diff --git a/.env.example b/.env.example index 2d8139a..03839aa 100644 --- a/.env.example +++ b/.env.example @@ -1 +1,6 @@ ANTHROPIC_API_KEY=sk-ant-your-api-key-here + +# Position and the watch set are hand-authored and live in your private repo, +# not this one. Defaults: ~/.config/insightweaver/{position,watches}.yaml +POSITION_PATH=~/.config/insightweaver/position.yaml +WATCHES_PATH=~/.config/insightweaver/watches.yaml diff --git a/.gitignore b/.gitignore index f887137..689343b 100644 --- a/.gitignore +++ b/.gitignore @@ -1,6 +1,17 @@ # Environment variables .env +# Position and the watch set (backlog task 013, 2026-08-31). +# These name real decisions, deadlines, exposures and beliefs. They belong in +# the operator's PRIVATE repo -- under version control there, because the +# quarterly review's value is the diff -- and never in this public one. Point +# POSITION_PATH / WATCHES_PATH at them; the defaults live outside any checkout. +# The *.example.yaml files in config/ are committed and are not matched here. +/position.yaml +/watches.yaml +config/position.yaml +config/watches.yaml + # User profile and personal configuration (contains personal information) config/user_profile.json config/context_modules/core/decision_context.json diff --git a/Makefile b/Makefile index e1e8db1..a42e67d 100644 --- a/Makefile +++ b/Makefile @@ -1,4 +1,4 @@ -.PHONY: help install install-dev test lint format fmt typecheck check pre-commit clean clean-all coverage db-drop-briefing update-deps +.PHONY: help install install-dev test lint format fmt typecheck check pre-commit clean clean-all coverage db-add-watches db-drop-briefing update-deps # Tool resolution (added 2026-08-24). # Before this, every recipe called bare `pytest` / `ruff` / `mypy`, which only @@ -44,6 +44,7 @@ help: @echo " make check Run all checks (lint + typecheck + test)" @echo "" @echo "Database:" + @echo " make db-add-watches Create the watches table (additive, safe)" @echo " make db-drop-briefing Drop the deleted briefing product's tables" @echo " (DESTRUCTIVE; captures to a dump first)" @echo "" @@ -113,6 +114,12 @@ clean-all: clean rm -rf venv/ rm -rf .venv/ +# Additive: creates the watches table if it is absent and prints that it did +# nothing if it is not. No --confirm, because nothing here can lose data; the +# --down direction can, and requires one. (2026-08-31, backlog task 013.) +db-add-watches: + $(PYTHON) -m src.database.migrations.add_watches_table + # Deliberately does NOT pass --confirm. The migration refuses without it, so # running this target prints what it would destroy and stops; the operator types # the real command themselves. A Makefile target that drops tables on `make` is diff --git a/backlog/022-open-questions.md b/backlog/022-open-questions.md index 332f36e..0654111 100644 --- a/backlog/022-open-questions.md +++ b/backlog/022-open-questions.md @@ -61,7 +61,17 @@ has no partial-failure story, and that migrating off it later is real work. **To answer:** either grant `pricing:GetProducts` and I will produce the figures, or choose on shape and accept that the cost comparison is unverified. -## Q2 -- Watch versus the existing Decision + Factor model. Blocks task 013. +## Q2 -- Watch versus the existing Decision + Factor model. Answered by events; confirm or overrule. + +**What task 013 built, 2026-08-31.** Task 012 landed first and dropped `decisions`, +`decision_factors` and `decision_evidence`, all at 0 rows -- so there was nothing left to evolve +and the question became "clean shape or resurrection". 013 built `watches` as a new table and put +Decision in **Position**, not in the database: a decision is an operator commitment held in a +hand-edited file under git, not a row the system writes. That leaves one model of the idea, not +two. `belief` is a float and `triggers` is structured, which is the part of the old model worth +not repeating. Overrule this and 013 is the task to revisit. + +### Original question The audit found the repository already models most of Watch, and has never used it: `decision_factors.what_would_update_me` is a trigger, `.current_state_note` is a belief state, @@ -81,7 +91,15 @@ choice on its merits. But it is a schema decision with migration consequences an in six months). I have no recommendation strong enough to make silently; the task file requires the implementer to state the rule, but if you have a preference it belongs here first. -## Q4 -- The `so_what` -> decision link. Blocks task 013. +## Q4 -- The `so_what` -> decision link. Built as the key; confirm or overrule. + +**What task 013 built, 2026-08-31.** The key, with the prose as an annotation, as the acceptance +assumed: `so_what` is a mapping of `decision` (a key into the Position's `decisions`) and +`because` (the sentence naming what changes). A watch whose `decision` names nothing in the +Position is rejected at load and not stored. The prose is still required and still non-empty -- +the key alone would make the invariant checkable and unreadable. + +### Original question Invariant 2 requires every Watch to name a specific decision. Should that be a **free-text `so_what` string**, or a **foreign key into the Position's decisions** with the prose as an diff --git a/config/position.example.yaml b/config/position.example.yaml new file mode 100644 index 0000000..1e6f8bf --- /dev/null +++ b/config/position.example.yaml @@ -0,0 +1,52 @@ +# Position -- EXAMPLE ONLY. +# +# Your real Position does not live in this repository. It names real decisions, +# real deadlines and real exposures, and this repo is public. Keep it in a +# private repo under git -- under git because the quarterly review's value is +# the diff -- and point POSITION_PATH at it: +# +# export POSITION_PATH=~/src/position/position.yaml +# +# The default, if you set nothing, is ~/.config/insightweaver/position.yaml. +# .gitignore in this repo refuses a real position.yaml, but the gitignore is a +# seatbelt, not the seat. +# +# The rule this file is written against: a Position holds *stakes*, not +# interests. A stake has a date by which you must have decided, and something +# that is worse if you decide wrong. If you cannot write the deadline, it is an +# interest -- interesting, possibly important, and not what this system is for. +# The loader refuses a decision with no deadline and warns, loudly, when the +# whole file runs past two pages, because that is what drift looks like. + +version: 1 + +# When you last sat down with this file. Task 024 turns this into the quarterly +# review loop; for now it is a date you can read. +reviewed: 2026-08-31 + +decisions: + - key: fedramp-authorization-renewal + name: Renew the platform's FedRAMP Moderate authorization, or let it lapse + deadline: 2027-03-31 + stake: > + Renewal costs roughly two engineer-quarters of continuous-monitoring work. + Letting it lapse closes the federal pipeline, which was 40% of last year's + revenue. The decision has to be made before the annual assessment window + opens, not after the invoice arrives. + + - key: hire-compliance-engineer + name: Hire a dedicated compliance engineer, or keep the work distributed + deadline: 2026-12-15 + stake: > + If continuous-monitoring scope grows, the distributed arrangement stops + working and the failure mode is a missed POA&M deadline nobody owns. + Hiring ahead of that costs a salary against uncertain need; hiring after + it costs an authorization. + + - key: second-cloud-region + name: Commit to a second cloud region for the government workload + deadline: 2027-01-31 + stake: > + A second region is the only way to meet the availability commitment in the + renewal, and standing one up takes a quarter. Deciding in February means + deciding not to. diff --git a/config/watches.example.yaml b/config/watches.example.yaml new file mode 100644 index 0000000..a037ca1 --- /dev/null +++ b/config/watches.example.yaml @@ -0,0 +1,94 @@ +# Watches -- EXAMPLE ONLY. +# +# Like Position, the real file lives in your private repo. Point WATCHES_PATH at +# it; the default is ~/.config/insightweaver/watches.yaml. +# +# A watch is one claim you have pre-registered, the decision it serves, and what +# would count as evidence. Nothing but this file creates a watch. There is no +# `watch add` command and there is no path from a model response to a row in the +# watches table -- the system never authors its own watches (invariant 6). +# +# Every field is required and none of them defaults: +# +# id slug; the name an alert will carry +# claim the thing that is or is not true, written so you could +# tell afterwards which it was +# belief your probability now, 0.0 to 1.0 +# so_what.decision a key from position.yaml -- not prose, a key, so that +# "which watches serve this decision" is a query +# so_what.because what changes for that decision if the claim resolves +# triggers structured; see below +# expires when the question stops mattering +# staleness_alert_days after how much silence the silence is itself the +# finding. Accepts a whole number of days or a cadence +# string ("30d", "2w", "3m", "1y"). +# +# ON TRIGGERS, which is the field people get wrong: +# +# `triggers` is a list of clauses, and a clause is a mapping over `terms`, +# `entities` and `sources`. It is NOT a sentence. Within a clause, every field +# you populate must match (AND) and any value within a field will do (OR); +# across clauses, any clause firing fires the watch. So the shape is a +# disjunction of conjunctions, which Tier 1 compiles into word-boundary regexes +# and a source-membership test with no model involved anywhere. +# +# The reason the loader rejects prose here rather than storing it: the deleted +# prediction ledger had a free-text `trigger_condition`, 33 rows were written +# against it, and none of them were ever graded, because nothing downstream +# could evaluate a sentence. A trigger you cannot compile is a trigger that +# never fires. + +version: 1 + +watches: + - id: fedramp-conmon-scope-expands + claim: > + FedRAMP expands continuous-monitoring requirements for Moderate systems in + a way that adds recurring work, before the 2027 assessment window. + belief: 0.35 + so_what: + decision: fedramp-authorization-renewal + because: > + Scope expansion moves renewal from two engineer-quarters to three or + more, which is the point at which lapsing becomes the cheaper option. + triggers: + - entities: [FedRAMP PMO, GSA] + terms: [continuous monitoring, ConMon, significant change] + - sources: [Federal Register] + terms: [FedRAMP] + expires: 2027-03-31 + staleness_alert_days: 30 + + - id: fedramp-20x-supersedes-rev5 + claim: > + The FedRAMP 20x program replaces the Rev 5 baseline for new + authorizations, rather than running alongside it. + belief: 0.45 + so_what: + decision: fedramp-authorization-renewal + because: > + If 20x supersedes rather than supplements, the renewal is a + re-authorization against a different baseline and the estimate is wrong + by a factor, not a margin. + triggers: + - terms: [FedRAMP 20x, Rev 5, revision 5] + entities: [FedRAMP PMO] + expires: 2027-06-30 + staleness_alert_days: 3w + + - id: compliance-hiring-market-tightens + claim: > + Median time-to-hire for a compliance engineer in the DC market exceeds 90 + days through the end of 2026. + belief: 0.6 + so_what: + decision: hire-compliance-engineer + because: > + A 90-day search started in December lands after the assessment window + opens, which converts "hire later if needed" into "do not hire". + triggers: + - terms: [time to hire, compliance engineer, GRC hiring] + - entities: [OPM] + terms: [cyber workforce, hiring authority] + expires: 2026-12-15 + staleness_alert_days: 45 diff --git a/src/cli/app.py b/src/cli/app.py index d828060..fc2ee4e 100644 --- a/src/cli/app.py +++ b/src/cli/app.py @@ -20,6 +20,7 @@ from .colors import accent, header, muted from .output import set_debug_mode from .sources import sources_command +from .watch import watch_command # Maps command prefix to its Click command object. # @@ -29,18 +30,22 @@ # the ordering constraint still applies to whatever is added next. COMMAND_DISPATCH = { "sources": sources_command, + "watch": watch_command, } def print_command_refresher(): """Print a short refresher of available commands.""" - click.echo(f"\n{header('Commands:')} {accent('sources')} | help | exit\n") + click.echo(f"\n{header('Commands:')} {accent('sources')} | {accent('watch')} | help | exit\n") def print_help(): """Print full help text for interactive mode.""" click.echo(header("Available commands:")) click.echo(f" {accent('sources')} - Inspect configured feeds and what they hold") + click.echo( + f" {accent('watch')} - Pre-registered watches and the decisions they serve" + ) click.echo(f" {accent('help')} - Show this help message") click.echo(f" {accent('exit')} - Exit InsightWeaver") click.echo() @@ -48,6 +53,17 @@ def print_help(): click.echo(f" {accent('sources list')} - All feeds with stored-article counts") click.echo(f" {accent('sources show')} - Detailed view for one feed") click.echo() + click.echo(header("Watch command:")) + click.echo( + f" {accent('watch list')} - Each watch, belief, decision, days to expiry" + ) + click.echo( + f" {accent('watch sync')} - Load the hand-authored Position and watch files" + ) + click.echo( + muted(" Watches are only ever created from those files -- there is no 'watch add'.") + ) + click.echo() click.echo(muted("Tip: Add --debug to any command to see detailed logs")) click.echo() @@ -154,10 +170,10 @@ def interactive_mode(): @click.version_option(version="1.0.0", prog_name="InsightWeaver") def cli(ctx, debug): """ - InsightWeaver - RSS feed ingestion and source inspection. + InsightWeaver - monitoring against pre-registered watches. The briefing product was removed in backlog task 012. What remains is - ingestion and the source layer. + ingestion, the source layer, and the Position/Watch units added by task 013. """ set_debug_mode(debug) ctx.ensure_object(dict) diff --git a/src/cli/watch.py b/src/cli/watch.py new file mode 100644 index 0000000..ca68e8e --- /dev/null +++ b/src/cli/watch.py @@ -0,0 +1,146 @@ +""" +Watch command - what is being monitored, and for which decision. + +Two subcommands and deliberately no third: + +``watch sync`` reads the hand-authored Position and watch files and makes the +table match them. It takes no arguments describing a watch, because it is not a +way to write one -- it is the file's only route into the database. + +``watch list`` shows what is stored: each watch, its belief, the decision it +serves, and how long until it expires. + +**There is no ``watch add``.** It would satisfy no acceptance criterion in +backlog task 013 and it is the seam through which auto-generated watches +arrive: once a command can take ``--claim`` and ``--belief`` from a caller, the +caller can be a model, and invariant 6 -- the system never authors its own +watches -- is then a convention rather than a property. The check that this +stays true is in ``tests/cli/test_watch_cli.py``, asserting on the registered +subcommands and their parameters rather than on anyone's intentions. + +Added 2026-08-31 for backlog task 013. +""" + +from datetime import date + +import click + +from ..database.connection import get_db +from ..database.models import Watch as WatchRow +from ..position import PositionError, WatchError, load_position, load_watches, sync_watches +from .colors import accent, error, header, muted, warning + + +def _load_from_files(today: date | None = None): + """ + Position and watches, validated together, or a ClickException naming why not. + + The two files are loaded as a pair because a watch is only valid against a + Position: ``so_what.decision`` has to name a decision that exists. + """ + try: + position = load_position(today=today) + watches = load_watches(position=position, today=today) + except FileNotFoundError as exc: + raise click.ClickException(str(exc)) + except (PositionError, WatchError) as exc: + raise click.ClickException(str(exc)) + return position, watches + + +def _echo_warnings(position) -> None: + """Print the Position's drift warnings. They warn; they never block.""" + for note in position.warnings: + click.echo(warning(f" ! {note}")) + + +def _expiry_phrase(expires: date, today: date) -> str: + days = (expires - today).days + if days < 0: + return f"{expires.isoformat()} (expired {-days}d ago)" + if days == 0: + return f"{expires.isoformat()} (expires today)" + return f"{expires.isoformat()} ({days}d to expiry)" + + +@click.group(name="watch") +def watch_command(): + """Pre-registered watches: what is monitored, and for which decision.""" + pass + + +@watch_command.command(name="sync") +def sync_watch_set(): + """Make the watches table match the hand-authored files.""" + position, watches = _load_from_files() + + click.echo(header("WATCH SYNC")) + click.echo(muted(f"position: {position.path}")) + click.echo(muted(f"watches: {watches[0].source_path if watches else 'none'}")) + if position.warnings: + click.echo() + click.echo(warning("Position warnings (nothing was rejected on these):")) + _echo_warnings(position) + click.echo() + + with get_db() as session: + summary = sync_watches(session, watches) + + for label in ("added", "updated", "removed"): + ids = summary[label] + if ids: + click.echo(f" {label}: {', '.join(sorted(ids))}") + if not any(summary.values()): + click.echo(muted(" no change")) + click.echo() + click.echo(muted(f"{len(watches)} watch(es) now match {position.path.name}'s decisions.")) + + +@watch_command.command(name="list") +def list_watches(): + """Show each watch, its belief, its decision, and days to expiry.""" + today = date.today() + + # The decision *names* come from Position, which may not be present on this + # machine -- it lives in a private repo. The keys are stored on the row, so + # the listing degrades to keys rather than failing. + names: dict[str, str] = {} + position_note = None + try: + position = load_position() + names = {d.key: d.name for d in position.decisions} + except (FileNotFoundError, PositionError) as exc: + position_note = str(exc).splitlines()[0] + + with get_db() as session: + rows = session.query(WatchRow).order_by(WatchRow.expires).all() + + if not rows: + click.echo(muted("No watches stored. Run 'watch sync' to load the checked-in files.")) + return + + click.echo(header("WATCHES")) + click.echo( + muted( + "Every watch names a decision. A watch that named none would not be " + "monitoring, and the loader refuses to store one." + ) + ) + if position_note: + click.echo(error(f"Position unreadable, showing decision keys only: {position_note}")) + click.echo("=" * 80) + + for row in rows: + click.echo(accent(row.id)) + click.echo(f" claim: {row.claim}") + click.echo(f" belief: {row.belief:.2f}") + decision = names.get(row.decision_key) + suffix = f" -- {decision}" if decision else "" + click.echo(f" decision: {row.decision_key}{suffix}") + click.echo(f" so what: {row.so_what}") + click.echo(f" expires: {_expiry_phrase(row.expires, today)}") + click.echo( + muted(f" stale after {row.staleness_alert_days}d of silence") + + muted(f" | {len(row.triggers or [])} trigger clause(s)") + ) + click.echo() diff --git a/src/config/settings.py b/src/config/settings.py index ef06fb8..fe5d30f 100644 --- a/src/config/settings.py +++ b/src/config/settings.py @@ -24,6 +24,20 @@ class Settings(BaseSettings): data_dir: Path = project_root / "data" logs_dir: Path = project_root / "src" / "logs" + # Position and the watch set are hand-authored and live in the operator's + # *private* repository -- they name real decisions, deadlines and exposures + # and this repo is public. The defaults deliberately point outside any + # checkout so that a missing configuration cannot resolve to a file in the + # tree. This repo carries config/position.example.yaml and + # config/watches.example.yaml, and .gitignore refuses the real ones. + # (2026-08-31, backlog task 013.) + position_path: Path = Path( + os.getenv("POSITION_PATH", "~/.config/insightweaver/position.yaml") + ).expanduser() + watches_path: Path = Path( + os.getenv("WATCHES_PATH", "~/.config/insightweaver/watches.yaml") + ).expanduser() + class Config: env_file = ".env" case_sensitive = False diff --git a/src/database/migrations/add_watches_table.py b/src/database/migrations/add_watches_table.py new file mode 100644 index 0000000..458044b --- /dev/null +++ b/src/database/migrations/add_watches_table.py @@ -0,0 +1,89 @@ +""" +Migration: add the ``watches`` table. + +Additive. It creates one table and touches nothing that exists -- unlike the +three drop migrations beside it, running this cannot lose data, so it needs no +``--confirm`` on the way up. + +The DDL is not written out here. The table is created from +``src.database.models.Watch.__table__``, so the migration and the model cannot +drift: a column added to the model is a column this migration creates, and there +is no second copy of the schema to forget to update. The CHECK constraints ride +along, which is the point -- ``ck_watches_so_what_present`` is invariant 2 +enforced at the storage layer, and a migration that quietly created the table +without it would leave the invariant resting on the loader alone. + +``downgrade()`` **does** destroy data: dropping the table discards whatever +watches were synced, and while every watch is reconstructible from the +operator's private ``watches.yaml``, the belief history that later tasks will +hang off these rows is not. So the way down requires ``--confirm`` and the way +up does not. + +Added 2026-08-31 for backlog task 013. +""" + +from __future__ import annotations + +import argparse +import sys + +from sqlalchemy import inspect +from sqlalchemy.engine import Engine + +from src.database.connection import engine +from src.database.models import Watch + +__all__ = ["TABLE", "downgrade", "upgrade"] + +TABLE = "watches" + +_CONFIRM_HELP = ( + "Dropping the watches table discards every synced watch and any state hung\n" + "off it. Re-run with --confirm if that is what you want:\n\n" + " python -m src.database.migrations.add_watches_table --down --confirm\n" +) + + +def upgrade(target: Engine | None = None) -> bool: + """Create ``watches`` if it is not already there. Returns True if it created it.""" + target = target or engine + if inspect(target).has_table(TABLE): + print(f"{TABLE} already exists; nothing to do.") + return False + Watch.__table__.create(bind=target) + print(f"Created {TABLE} ({len(Watch.__table__.columns)} columns).") + return True + + +def downgrade(target: Engine | None = None, *, confirmed: bool = False) -> bool: + """Drop ``watches``. Requires ``confirmed`` -- this loses rows.""" + if not confirmed: + raise SystemExit(_CONFIRM_HELP) + target = target or engine + if not inspect(target).has_table(TABLE): + print(f"{TABLE} does not exist; nothing to do.") + return False + Watch.__table__.drop(bind=target) + print(f"Dropped {TABLE}.") + return True + + +def main(argv: list[str] | None = None) -> int: + parser = argparse.ArgumentParser( + prog="add_watches_table", + description=__doc__, + formatter_class=argparse.RawDescriptionHelpFormatter, + ) + parser.add_argument("--down", action="store_true", help="drop the table again") + parser.add_argument("--confirm", action="store_true", help="required by --down") + args = parser.parse_args(argv) + + if args.down: + downgrade(confirmed=args.confirm) + return 0 + upgrade() + return 0 + + +if __name__ == "__main__": + sys.exit(main()) diff --git a/src/database/models.py b/src/database/models.py index 4eac0be..357ac4b 100644 --- a/src/database/models.py +++ b/src/database/models.py @@ -1,7 +1,8 @@ """ Database models. -Two tables: the feeds we read and the articles we stored from them. +Three tables: the feeds we read, the articles we stored from them, and the +watches the operator has pre-registered. Everything else was deleted by backlog task 012 along with the briefing product that owned it -- syntheses, context snapshots, provenance, topic clusters, @@ -13,12 +14,17 @@ ``articles`` is deliberately untouched. It holds the corpus (55,249 rows as of 2026-08-31) and what becomes of it is backlog task 014's decision, not this one's. + +``watches`` is the first table of the rewrite, added 2026-08-31 by backlog task +013. Its CHECK constraints are not decoration -- see the class docstring. """ from sqlalchemy import ( JSON, Boolean, + CheckConstraint, Column, + Date, DateTime, Float, ForeignKey, @@ -114,3 +120,61 @@ class Article(Base): Index("idx_articles_filtered_fetched", "filtered", "fetched_at"), Index("idx_articles_filtered_published", "filtered", "published_date"), ) + + +class Watch(Base): + """ + One pre-registered claim, the decision it serves, and what would move it. + + Rows arrive from exactly one place: ``src.position.watches.sync_watches``, + reading the operator's hand-authored file. Nothing else writes here. + Invariant 6 -- the system never authors its own watches -- is a property of + the code paths that exist, so no path exists. + + **The constraints are the enforcement, not a formality.** Invariant 2 says + every Watch must name a decision; the loader rejects a blank ``so_what``, + and ``ck_watches_so_what_present`` rejects it again at the storage layer, so + a row written by hand through ``sqlite3`` fails the same way a bad YAML file + does. The repository's own history is the argument: 25 unfalsifiable + predictions accumulated in the deleted ledger because a missing field was + tolerated by every layer that could have refused it. + + ``triggers`` is JSON, and structured JSON specifically -- a list of clauses + over terms, entities and source allowlists. Tier 1 compiles it into a + deterministic predicate. The column it replaces, ``predictions. + trigger_condition``, was free text; 33 rows were written against it and none + were ever graded, because nothing downstream could evaluate a sentence. + """ + + __tablename__ = "watches" + + # The operator's own id from the file, not a surrogate. It is the name they + # will type and the name an alert will carry, and a watch that is renamed in + # the file is a different watch. + id = Column(String(100), primary_key=True) + + claim = Column(Text, nullable=False) + belief = Column(Float, nullable=False) + + # so_what, split: the key is what makes invariant 2 machine-checkable, the + # prose is what makes it readable. `decision_key` references a decision in + # the Position file, which is not a table -- Position lives in a private + # repo under git, so this is deliberately not a foreign key. + decision_key = Column(String(100), nullable=False) + so_what = Column(Text, nullable=False) + + triggers = Column(JSON, nullable=False) + expires = Column(Date, nullable=False) + staleness_alert_days = Column(Integer, nullable=False) + + created_at = Column(DateTime, default=utcnow) + updated_at = Column(DateTime, default=utcnow, onupdate=utcnow) + + __table_args__ = ( + CheckConstraint("length(trim(so_what)) > 0", name="ck_watches_so_what_present"), + CheckConstraint("length(trim(decision_key)) > 0", name="ck_watches_decision_present"), + CheckConstraint("belief >= 0.0 AND belief <= 1.0", name="ck_watches_belief_range"), + CheckConstraint("staleness_alert_days >= 1", name="ck_watches_staleness_min"), + Index("idx_watches_decision_key", "decision_key"), + Index("idx_watches_expires", "expires"), + ) diff --git a/src/position/__init__.py b/src/position/__init__.py new file mode 100644 index 0000000..e8228e8 --- /dev/null +++ b/src/position/__init__.py @@ -0,0 +1,42 @@ +""" +Position and Watch: the atomic units of the monitoring system. + +Position holds the decisions the operator is carrying. A Watch is one +pre-registered claim serving one of those decisions. Both are hand-authored +YAML in a private repository; this package is the schema, the validator and the +one path by which a Watch reaches the database. + +Added 2026-08-31 for backlog task 013. +""" + +from .position import ( + MAX_PAGES, + POSITION_PAGE_WORDS, + Decision, + Position, + PositionError, + load_position, +) +from .watches import ( + TRIGGER_FIELDS, + TriggerClause, + Watch, + WatchError, + load_watches, + sync_watches, +) + +__all__ = [ + "MAX_PAGES", + "POSITION_PAGE_WORDS", + "TRIGGER_FIELDS", + "Decision", + "Position", + "PositionError", + "TriggerClause", + "Watch", + "WatchError", + "load_position", + "load_watches", + "sync_watches", +] diff --git a/src/position/position.py b/src/position/position.py new file mode 100644 index 0000000..96c64f1 --- /dev/null +++ b/src/position/position.py @@ -0,0 +1,318 @@ +""" +Position: the decisions the operator is actually carrying, with deadlines. + +Position is the root of the commitment graph. Everything downstream -- watches, +routing, adjudication, alerts -- exists to serve a decision named here. A watch +that serves no decision in this file is curiosity, and invariant 2 exists to +keep curiosity out of the notification path. + +**Position is not in this repository.** It names real decisions, real deadlines +and real exposures; this repo is public. The file lives in a private repo under +version control -- under version control because the quarterly review's value is +the *diff* (what did I believe last quarter, which decisions closed), which a +gitignored file cannot provide. What ships here is the schema, the loader and +``config/position.example.yaml``. The path is configurable +(``POSITION_PATH``) and defaults to ``~/.config/insightweaver/position.yaml``, +outside any checkout. + +**Why the size check warns and does not fail.** The brief says a Position over +two pages has drifted into describing interests rather than stakes. A hard limit +would enforce a proxy: a three-page Position full of genuine deadlines is better +than a one-page Position of vague interests. So the check names the drift and +lets the human judge, the same way ``coverage_probes`` reports INCONCLUSIVE +rather than dropping a probe. Everything that is *structurally* checkable -- +a decision with no deadline, a duplicate key, a date that is not a date -- is a +hard rejection, because those are not judgement calls. + +Added 2026-08-31 for backlog task 013. +""" + +from __future__ import annotations + +import logging +import re +from dataclasses import dataclass +from datetime import date, datetime +from pathlib import Path +from typing import Any + +import yaml + +__all__ = [ + "MAX_PAGES", + "POSITION_PAGE_WORDS", + "Decision", + "Position", + "PositionError", + "load_position", +] + +logger = logging.getLogger(__name__) + +# A page of prose, for the purpose of the drift warning only. The number is a +# proxy and is treated as one: it can only produce a warning, never a rejection. +POSITION_PAGE_WORDS = 500 +MAX_PAGES = 2 + +# Decision keys are slugs because watches reference them by key and a key with +# spaces or capitals invites two spellings of the same decision. +_KEY_RE = re.compile(r"^[a-z0-9][a-z0-9._-]*$") + +_TOP_LEVEL_KEYS = {"version", "reviewed", "decisions"} +_DECISION_KEYS = {"key", "name", "deadline", "stake"} + + +class PositionError(ValueError): + """ + Raised when a Position file cannot be read as a Position. + + Carries every problem found, not just the first: the file is hand-edited, + and a loader that reports one error per run turns a five-minute fix into + five runs. + """ + + def __init__(self, path: Path, problems: list[str]) -> None: + self.path = path + self.problems = list(problems) + body = "\n".join(f" - {p}" for p in self.problems) + super().__init__(f"{path} is not a valid Position:\n{body}") + + +@dataclass(frozen=True) +class Decision: + """One standing decision, with the date by which it stops being open.""" + + key: str + name: str + deadline: date + stake: str | None = None + + def days_remaining(self, today: date) -> int: + return (self.deadline - today).days + + +@dataclass(frozen=True) +class Position: + """A loaded Position, plus whatever the loader wants to say about it.""" + + path: Path + version: int + decisions: tuple[Decision, ...] + reviewed: date | None = None + warnings: tuple[str, ...] = () + word_count: int = 0 + + @property + def decision_keys(self) -> frozenset[str]: + return frozenset(d.key for d in self.decisions) + + def decision(self, key: str) -> Decision | None: + """The decision with this key, or None. Watches resolve `so_what` here.""" + for candidate in self.decisions: + if candidate.key == key: + return candidate + return None + + +def default_position_path() -> Path: + """Where Position lives when nothing says otherwise.""" + from src.config.settings import settings + + return Path(settings.position_path).expanduser() + + +def _as_date(value: Any) -> date | None: + """A YAML scalar as a date, or None if it is not one.""" + if isinstance(value, datetime): + return value.date() + if isinstance(value, date): + return value + if isinstance(value, str): + try: + return date.fromisoformat(value.strip()) + except ValueError: + return None + return None + + +def _is_blank(value: Any) -> bool: + """True for None, for a non-string, or for a string that is only whitespace.""" + return not isinstance(value, str) or not value.strip() + + +def _content_words(text: str) -> int: + """ + Words in the file, ignoring comments and blank lines. + + Comments are the operator's own notes about the decisions and counting them + would make the drift warning fire on a well-annotated file, which is the + opposite of what it is for. + """ + kept = [ + line for line in text.splitlines() if line.strip() and not line.lstrip().startswith("#") + ] + return len(" ".join(kept).split()) + + +def _validate_decision( + index: int, raw: Any, seen: set[str], problems: list[str] +) -> Decision | None: + label = f"decisions[{index}]" + if not isinstance(raw, dict): + problems.append( + f"{label}: expected a mapping with key/name/deadline, got {type(raw).__name__}" + ) + return None + + unknown = sorted(set(raw) - _DECISION_KEYS) + if unknown: + problems.append(f"{label}: unknown field(s) {unknown}; allowed: {sorted(_DECISION_KEYS)}") + + key = raw.get("key") + if _is_blank(key): + problems.append(f"{label}: 'key' is required and must be a non-empty string") + key = None + elif not _KEY_RE.match(str(key).strip()): + problems.append( + f"{label}: key '{key}' is not a slug -- lowercase letters, digits, '-', '_' and '.'" + ) + key = None + else: + key = str(key).strip() + if key in seen: + problems.append(f"{label}: duplicate decision key '{key}'") + key = None + else: + seen.add(key) + + name = raw.get("name") + if _is_blank(name): + problems.append(f"{label}: 'name' is required and must be a non-empty string") + + deadline = _as_date(raw.get("deadline")) + if "deadline" not in raw: + problems.append( + f"{label}: 'deadline' is required. A decision with no date is an interest; " + f"Position holds stakes." + ) + elif deadline is None: + problems.append(f"{label}: deadline {raw.get('deadline')!r} is not a date (YYYY-MM-DD)") + + stake = raw.get("stake") + + if key is None or _is_blank(name) or deadline is None: + return None + return Decision( + key=key, + name=str(name).strip(), + deadline=deadline, + stake=str(stake).strip() if isinstance(stake, str) and stake.strip() else None, + ) + + +def _drift_warnings(decisions: tuple[Decision, ...], word_count: int, today: date) -> list[str]: + """Everything the loader wants to say but will not refuse over.""" + notes: list[str] = [] + + limit = POSITION_PAGE_WORDS * MAX_PAGES + if word_count > limit: + pages = word_count / POSITION_PAGE_WORDS + notes.append( + f"Position is {word_count} words (~{pages:.1f} pages), over the {MAX_PAGES}-page mark. " + f"A Position this long has usually drifted from describing stakes -- decisions with " + f"deadlines and exposure -- into describing interests. Nothing is rejected on length; " + f"read it and cut what you are merely curious about." + ) + + for decision in decisions: + if decision.deadline < today: + notes.append( + f"decision '{decision.key}' passed its deadline " + f"({decision.deadline.isoformat()}, {(today - decision.deadline).days}d ago). " + f"Close it or move the date; watches serving a dead decision are noise." + ) + if decision.stake is None: + notes.append( + f"decision '{decision.key}' states no 'stake'. Without the exposure written down " + f"it reads as an interest, and there is nothing to weigh an alert against." + ) + return notes + + +def load_position(path: str | Path | None = None, *, today: date | None = None) -> Position: + """ + Read and validate a Position file. + + Raises :class:`FileNotFoundError` if the file is absent -- there is no + default Position, because a guessed set of decisions is worse than none -- + and :class:`PositionError` listing every structural problem otherwise. + Judgement-call problems come back on :attr:`Position.warnings` and are also + logged at WARNING. + """ + target = Path(path).expanduser() if path is not None else default_position_path() + if not target.exists(): + raise FileNotFoundError( + f"No Position at {target}. Position lives in your private repo, not this one; " + f"point POSITION_PATH at it, or start from config/position.example.yaml." + ) + + text = target.read_text(encoding="utf-8") + try: + raw = yaml.safe_load(text) + except yaml.YAMLError as exc: + raise PositionError(target, [f"file is not valid YAML: {exc}"]) + + problems: list[str] = [] + if raw is None: + raise PositionError(target, ["file is empty"]) + if not isinstance(raw, dict): + raise PositionError( + target, [f"expected a mapping at the top level, got {type(raw).__name__}"] + ) + + unknown = sorted(set(raw) - _TOP_LEVEL_KEYS) + if unknown: + problems.append(f"unknown top-level field(s) {unknown}; allowed: {sorted(_TOP_LEVEL_KEYS)}") + + version = raw.get("version", 1) + if not isinstance(version, int) or isinstance(version, bool): + problems.append(f"version must be an integer, got {version!r}") + version = 1 + + reviewed = None + if raw.get("reviewed") is not None: + reviewed = _as_date(raw.get("reviewed")) + if reviewed is None: + problems.append(f"reviewed {raw.get('reviewed')!r} is not a date (YYYY-MM-DD)") + + raw_decisions = raw.get("decisions") + decisions: list[Decision] = [] + if raw_decisions is None: + problems.append("'decisions' is required -- a Position with no decisions holds no stakes") + elif not isinstance(raw_decisions, list): + problems.append(f"'decisions' must be a list, got {type(raw_decisions).__name__}") + elif not raw_decisions: + problems.append("'decisions' is empty -- a Position with no decisions holds no stakes") + else: + seen: set[str] = set() + for index, entry in enumerate(raw_decisions): + decision = _validate_decision(index, entry, seen, problems) + if decision is not None: + decisions.append(decision) + + if problems: + raise PositionError(target, problems) + + word_count = _content_words(text) + notes = _drift_warnings(tuple(decisions), word_count, today or date.today()) + for note in notes: + logger.warning("position: %s", note) + + return Position( + path=target, + version=version, + decisions=tuple(decisions), + reviewed=reviewed, + warnings=tuple(notes), + word_count=word_count, + ) diff --git a/src/position/watches.py b/src/position/watches.py new file mode 100644 index 0000000..9fd46fd --- /dev/null +++ b/src/position/watches.py @@ -0,0 +1,583 @@ +""" +Watch: one pre-registered claim, the decision it serves, and what would move it. + +A Watch is the atomic unit of the monitoring system. It says what the operator +believes (``claim`` + ``belief``), which decision that belief bears on +(``so_what``), what evidence would be candidate evidence (``triggers``), when +the question stops mattering (``expires``), and how long silence is allowed to +run before silence itself is the finding (``staleness_alert_days``). + +Three shape decisions, each made against a specific thing this repository has +already got wrong: + +**Watch is a new table, not an evolution of Decision + Factor.** The audit found +``decision_factors.what_would_update_me`` was a trigger, ``.current_state_note`` +a belief state, and ``decisions.name`` what ``so_what`` points at -- all of it +holding zero rows, all of it dropped by task 012. So there is nothing to evolve: +the choice is not between two live models but between a clean shape and a +resurrection. What is worth carrying forward is why the old one stayed empty -- +it was prose all the way down, so nothing could be computed from it and nothing +ever was. ``belief`` is a float here and ``triggers`` is structured for exactly +that reason. Decision does survive, in Position, where it belongs: it is an +operator commitment, not a database row the system writes. + +**``so_what`` is a key plus prose, not prose alone.** It is a mapping of +``decision`` (a key into the Position's decisions) and ``because`` (the sentence +naming what changes). The key makes invariant 2 machine-checkable and makes +"which watches serve this decision" a query; the prose is what makes the answer +readable. An invariant enforced only by prose is the pattern the repository +spent the week correcting -- the ledger accumulated 25 unfalsifiable predictions +because a field was tolerated empty. + +**``triggers`` is structured and must stay that way.** It is a list of clauses +over ``terms``, ``entities`` and ``sources`` -- the three things Tier 1 can +compile into a deterministic word-boundary predicate. A prose trigger cannot be +compiled, and the repository has the receipt: 33 predictions were written +against a free-text ``trigger_condition`` and graded zero times. A clause that +is a sentence is rejected here, loudly, by name. + +The file itself lives beside Position in the private repo, for the same reason: +a watch names a real claim about a real exposure. This repo carries the schema +and ``config/watches.example.yaml``. + +Nothing in this module creates a Watch from anything but the file. There is no +constructor path from a model response, a candidate, or a CLI argument, because +invariant 6 says the system never authors its own watches, and the way that +invariant is lost is one convenient write path at a time. + +Added 2026-08-31 for backlog task 013. +""" + +from __future__ import annotations + +import logging +import re +from dataclasses import dataclass, field +from datetime import date +from pathlib import Path +from typing import Any + +import yaml + +from src.utils.cadence import InvalidCadence, parse_cadence + +from .position import Position, _as_date, _is_blank + +__all__ = [ + "TRIGGER_FIELDS", + "TriggerClause", + "Watch", + "WatchError", + "load_watches", + "sync_watches", +] + +logger = logging.getLogger(__name__) + +_ID_RE = re.compile(r"^[a-z0-9][a-z0-9._-]*$") + +_TOP_LEVEL_KEYS = {"version", "watches"} +_WATCH_KEYS = { + "id", + "claim", + "belief", + "so_what", + "triggers", + "expires", + "staleness_alert_days", +} +_SO_WHAT_KEYS = {"decision", "because"} + +# The only three axes a trigger clause may constrain. Each maps onto something +# Tier 1 can evaluate without a model: a word-boundary term match, an entity's +# declared surface forms, and the adapter that produced the observation. +TRIGGER_FIELDS = ("terms", "entities", "sources") + + +class WatchError(ValueError): + """ + Raised when a watches file cannot be read as watches. + + Like :class:`~src.position.position.PositionError`, it carries every problem + found rather than the first, and it is raised **before anything is written**: + a file with one bad watch stores none of its watches, so the table never + holds a half-applied file. + """ + + def __init__(self, path: Path, problems: list[str]) -> None: + self.path = path + self.problems = list(problems) + body = "\n".join(f" - {p}" for p in self.problems) + super().__init__(f"{path} is not a valid watch set:\n{body}") + + +@dataclass(frozen=True) +class TriggerClause: + """ + One conjunctive clause of a trigger. + + A clause fires when every field it populates matches: any of its ``terms`` + appears, **and** any of its ``entities`` appears, **and** the observation + came from one of its ``sources``. Fields left out are not constraints. The + watch fires when any clause fires, so ``triggers`` as a whole is a + disjunction of conjunctions -- expressive enough to say "FedRAMP *and* + continuous monitoring, or anything about it in the Federal Register", and + small enough that Tier 1 compiles it to regexes and a set membership test + with no interpreter in between. + """ + + terms: tuple[str, ...] = () + entities: tuple[str, ...] = () + sources: tuple[str, ...] = () + + def as_dict(self) -> dict[str, list[str]]: + """The JSON shape stored in ``watches.triggers``. Empty fields are omitted.""" + return {name: list(getattr(self, name)) for name in TRIGGER_FIELDS if getattr(self, name)} + + +@dataclass(frozen=True) +class Watch: + """One pre-registered claim. Constructed only by :func:`load_watches`.""" + + id: str + claim: str + belief: float + decision_key: str + so_what: str + triggers: tuple[TriggerClause, ...] + expires: date + staleness_alert_days: int + source_path: Path | None = field(default=None, compare=False) + + def days_to_expiry(self, today: date) -> int: + return (self.expires - today).days + + def triggers_json(self) -> list[dict[str, list[str]]]: + return [clause.as_dict() for clause in self.triggers] + + +def default_watches_path() -> Path: + """Where the watch set lives when nothing says otherwise.""" + from src.config.settings import settings + + return Path(settings.watches_path).expanduser() + + +def _validate_so_what( + label: str, raw: Any, position: Position, problems: list[str] +) -> tuple[str, str] | None: + """ + Invariant 2, enforced. Returns ``(decision_key, because)`` or None. + + Every branch here rejects. None of them defaults, substitutes a placeholder, + or falls back to the claim text -- a Watch with no articulable decision is + curiosity, and the point of the invariant is that curiosity does not get + stored and then quietly alert. + """ + if raw is None: + problems.append( + f"{label}: 'so_what' is required and names the decision this watch serves. " + f"A watch with no decision attached is curiosity, not monitoring." + ) + return None + if isinstance(raw, str): + if not raw.strip(): + problems.append( + f"{label}: 'so_what' is empty. It must name a decision from the Position and " + f"say what changes; blank is rejected, never defaulted." + ) + else: + problems.append( + f"{label}: 'so_what' must be a mapping of 'decision' (a key from the Position) " + f"and 'because' (what changes), not a bare string." + ) + return None + if not isinstance(raw, dict): + problems.append(f"{label}: 'so_what' must be a mapping, got {type(raw).__name__}") + return None + + unknown = sorted(set(raw) - _SO_WHAT_KEYS) + if unknown: + problems.append( + f"{label}: so_what has unknown field(s) {unknown}; allowed: ['because', 'decision']" + ) + + decision_key = raw.get("decision") + because = raw.get("because") + ok = True + + if _is_blank(decision_key): + problems.append( + f"{label}: so_what.decision is required and must name a decision key from the " + f"Position. Known keys: {sorted(position.decision_keys)}" + ) + ok = False + elif str(decision_key).strip() not in position.decision_keys: + problems.append( + f"{label}: so_what.decision '{str(decision_key).strip()}' names no decision in " + f"{position.path}. Known keys: {sorted(position.decision_keys)}. Add the decision " + f"to the Position, or drop the watch." + ) + ok = False + + if _is_blank(because): + problems.append( + f"{label}: so_what.because is required and must be a non-empty sentence saying what " + f"changes for that decision if the claim resolves." + ) + ok = False + + if not ok: + return None + return str(decision_key).strip(), str(because).strip() + + +def _validate_string_list( + label: str, field_name: str, raw: Any, problems: list[str] +) -> tuple[str, ...]: + if raw is None: + return () + if isinstance(raw, str): + problems.append( + f"{label}: trigger '{field_name}' must be a list of terms, not a string. " + f"A trigger is compiled into a predicate, so it cannot be a sentence." + ) + return () + if not isinstance(raw, list): + problems.append(f"{label}: trigger '{field_name}' must be a list, got {type(raw).__name__}") + return () + values: list[str] = [] + for item in raw: + if _is_blank(item): + problems.append(f"{label}: trigger '{field_name}' contains an empty entry") + continue + values.append(str(item).strip()) + return tuple(values) + + +def _validate_triggers(label: str, raw: Any, problems: list[str]) -> tuple[TriggerClause, ...]: + """ + Structure, not prose. Every rejection here is a rejection of prose. + + ``trigger_condition`` in the deleted ledger accepted free text, and 33 rows + were written against it and graded zero times, because nothing downstream + could evaluate a sentence. Tier 1 compiles these clauses; a clause it cannot + compile is refused at the door rather than stored and skipped. + """ + if raw is None: + problems.append( + f"{label}: 'triggers' is required -- what observation would be candidate evidence?" + ) + return () + if isinstance(raw, str): + problems.append( + f"{label}: 'triggers' is prose. It must be a list of clauses over " + f"{list(TRIGGER_FIELDS)}, because Tier 1 compiles triggers into a deterministic " + f"predicate and cannot compile a sentence." + ) + return () + if not isinstance(raw, list): + problems.append(f"{label}: 'triggers' must be a list, got {type(raw).__name__}") + return () + if not raw: + problems.append(f"{label}: 'triggers' is empty -- a watch nothing can trigger never fires") + return () + + clauses: list[TriggerClause] = [] + for index, entry in enumerate(raw): + clause_label = f"{label}.triggers[{index}]" + if isinstance(entry, str): + problems.append( + f"{clause_label}: a trigger clause is a mapping over {list(TRIGGER_FIELDS)}, " + f"not the sentence {entry.strip()[:60]!r}. Prose triggers are what left 33 " + f"predictions ungradeable; write the terms and entities that would match." + ) + continue + if not isinstance(entry, dict): + problems.append(f"{clause_label}: expected a mapping, got {type(entry).__name__}") + continue + unknown = sorted(set(entry) - set(TRIGGER_FIELDS)) + if unknown: + problems.append( + f"{clause_label}: unknown field(s) {unknown}; a clause may constrain only " + f"{list(TRIGGER_FIELDS)}" + ) + continue + clause = TriggerClause( + terms=_validate_string_list(clause_label, "terms", entry.get("terms"), problems), + entities=_validate_string_list( + clause_label, "entities", entry.get("entities"), problems + ), + sources=_validate_string_list(clause_label, "sources", entry.get("sources"), problems), + ) + if not (clause.terms or clause.entities or clause.sources): + problems.append( + f"{clause_label}: constrains nothing. A clause with no terms, entities or " + f"sources matches every observation." + ) + continue + clauses.append(clause) + return tuple(clauses) + + +def _validate_belief(label: str, raw: Any, problems: list[str]) -> float | None: + if raw is None: + problems.append(f"{label}: 'belief' is required -- a probability in [0.0, 1.0]") + return None + if isinstance(raw, bool) or not isinstance(raw, int | float): + problems.append(f"{label}: belief must be a number in [0.0, 1.0], got {raw!r}") + return None + value = float(raw) + if not (0.0 <= value <= 1.0): + problems.append(f"{label}: belief {value} is outside [0.0, 1.0]") + return None + return value + + +def _validate_staleness(label: str, raw: Any, problems: list[str]) -> int | None: + """ + Days, written as an integer or as a cadence (``2w``, ``30d``). + + The cadence spelling goes through :mod:`src.utils.cadence` rather than a + second parser, so the whole system reads one interval grammar. + """ + if raw is None: + problems.append( + f"{label}: 'staleness_alert_days' is required -- after how much silence is the " + f"silence itself the finding?" + ) + return None + if isinstance(raw, bool): + problems.append( + f"{label}: staleness_alert_days must be a whole number of days, got {raw!r}" + ) + return None + if isinstance(raw, str): + try: + days = parse_cadence(raw).days + except InvalidCadence as exc: + problems.append(f"{label}: staleness_alert_days {raw!r} is not an interval -- {exc}") + return None + elif isinstance(raw, int): + days = raw + else: + problems.append( + f"{label}: staleness_alert_days must be a whole number of days, got {raw!r}" + ) + return None + if days < 1: + problems.append( + f"{label}: staleness_alert_days is {days}; it must be at least 1. Zero would make " + f"every watch stale the moment it is loaded." + ) + return None + return days + + +def _validate_watch( + index: int, + raw: Any, + position: Position, + seen: set[str], + today: date, + problems: list[str], +) -> Watch | None: + label = f"watches[{index}]" + if not isinstance(raw, dict): + problems.append(f"{label}: expected a mapping, got {type(raw).__name__}") + return None + + watch_id = raw.get("id") + if _is_blank(watch_id): + problems.append(f"{label}: 'id' is required and must be a non-empty string") + watch_id = None + elif not _ID_RE.match(str(watch_id).strip()): + problems.append( + f"{label}: id '{watch_id}' is not a slug -- lowercase letters, digits, '-', '_', '.'" + ) + watch_id = None + else: + watch_id = str(watch_id).strip() + if watch_id in seen: + problems.append(f"{label}: duplicate watch id '{watch_id}'") + watch_id = None + else: + seen.add(watch_id) + + label = f"watches[{index}]" if watch_id is None else f"watch '{watch_id}'" + + unknown = sorted(set(raw) - _WATCH_KEYS) + if unknown: + problems.append(f"{label}: unknown field(s) {unknown}; allowed: {sorted(_WATCH_KEYS)}") + + claim = raw.get("claim") + if _is_blank(claim): + problems.append(f"{label}: 'claim' is required and must be a non-empty string") + + belief = _validate_belief(label, raw.get("belief"), problems) + so_what = _validate_so_what(label, raw.get("so_what"), position, problems) + triggers = _validate_triggers(label, raw.get("triggers"), problems) + staleness = _validate_staleness(label, raw.get("staleness_alert_days"), problems) + + expires = None + if "expires" not in raw or raw.get("expires") is None: + problems.append( + f"{label}: 'expires' is required -- a watch with no end date outlives the decision " + f"it serves and becomes a subscription." + ) + else: + expires = _as_date(raw.get("expires")) + if expires is None: + problems.append(f"{label}: expires {raw.get('expires')!r} is not a date (YYYY-MM-DD)") + elif expires < today: + problems.append( + f"{label}: expires {expires.isoformat()} is in the past " + f"({(today - expires).days}d ago). Extend it or delete the watch." + ) + expires = None + + if ( + watch_id is None + or _is_blank(claim) + or belief is None + or so_what is None + or not triggers + or expires is None + or staleness is None + ): + return None + + decision_key, because = so_what + return Watch( + id=watch_id, + claim=str(claim).strip(), + belief=belief, + decision_key=decision_key, + so_what=because, + triggers=triggers, + expires=expires, + staleness_alert_days=staleness, + ) + + +def load_watches( + path: str | Path | None = None, + *, + position: Position, + today: date | None = None, +) -> list[Watch]: + """ + Read and validate the watch set against a loaded Position. + + Raises :class:`FileNotFoundError` if the file is absent and + :class:`WatchError` listing every problem otherwise. It never returns a + partially valid set: if any watch is rejected, the call raises and the + caller has nothing to store. + """ + target = Path(path).expanduser() if path is not None else default_watches_path() + if not target.exists(): + raise FileNotFoundError( + f"No watch set at {target}. Watches live beside Position in your private repo; " + f"point WATCHES_PATH at it, or start from config/watches.example.yaml." + ) + + try: + raw = yaml.safe_load(target.read_text(encoding="utf-8")) + except yaml.YAMLError as exc: + raise WatchError(target, [f"file is not valid YAML: {exc}"]) + + if raw is None: + raise WatchError(target, ["file is empty"]) + if not isinstance(raw, dict): + raise WatchError(target, [f"expected a mapping at the top level, got {type(raw).__name__}"]) + + problems: list[str] = [] + unknown = sorted(set(raw) - _TOP_LEVEL_KEYS) + if unknown: + problems.append(f"unknown top-level field(s) {unknown}; allowed: {sorted(_TOP_LEVEL_KEYS)}") + + entries = raw.get("watches") + watches: list[Watch] = [] + if entries is None: + problems.append("'watches' is required") + elif not isinstance(entries, list): + problems.append(f"'watches' must be a list, got {type(entries).__name__}") + else: + seen: set[str] = set() + stamp = today or date.today() + for index, entry in enumerate(entries): + watch = _validate_watch(index, entry, position, seen, stamp, problems) + if watch is not None: + watches.append(watch) + + if problems: + raise WatchError(target, problems) + + return [ + Watch( + id=w.id, + claim=w.claim, + belief=w.belief, + decision_key=w.decision_key, + so_what=w.so_what, + triggers=w.triggers, + expires=w.expires, + staleness_alert_days=w.staleness_alert_days, + source_path=target, + ) + for w in watches + ] + + +def sync_watches(session: Any, watches: list[Watch]) -> dict[str, list[str]]: + """ + Make the ``watches`` table match the file, and report what moved. + + The file is authoritative: a watch removed from it is removed from the + table. That is the whole write path -- there is no other way a row gets into + this table, and there is deliberately no argument by which a caller could + describe a watch that is not in the file. + + Takes already-validated :class:`Watch` values, so an invalid file cannot + reach this function at all. + """ + from src.database.models import Watch as WatchRow + + incoming = {w.id: w for w in watches} + existing = {row.id: row for row in session.query(WatchRow).all()} + + summary: dict[str, list[str]] = {"added": [], "updated": [], "removed": []} + + for watch_id, watch in incoming.items(): + row = existing.get(watch_id) + if row is None: + session.add( + WatchRow( + id=watch.id, + claim=watch.claim, + belief=watch.belief, + so_what=watch.so_what, + decision_key=watch.decision_key, + triggers=watch.triggers_json(), + expires=watch.expires, + staleness_alert_days=watch.staleness_alert_days, + ) + ) + summary["added"].append(watch_id) + continue + row.claim = watch.claim + row.belief = watch.belief + row.so_what = watch.so_what + row.decision_key = watch.decision_key + row.triggers = watch.triggers_json() + row.expires = watch.expires + row.staleness_alert_days = watch.staleness_alert_days + summary["updated"].append(watch_id) + + for watch_id, row in existing.items(): + if watch_id not in incoming: + session.delete(row) + summary["removed"].append(watch_id) + + session.flush() + return summary diff --git a/tests/cli/test_app.py b/tests/cli/test_app.py index b6d6de3..5dff87e 100644 --- a/tests/cli/test_app.py +++ b/tests/cli/test_app.py @@ -18,7 +18,7 @@ def test_cli_help(self, cli_runner): assert result.exit_code == 0 assert "InsightWeaver" in result.output - assert "RSS feed ingestion" in result.output + assert "monitoring against pre-registered watches" in result.output def test_cli_version(self, cli_runner): """Should show version""" @@ -59,6 +59,23 @@ def test_sources_command_registered(self, cli_runner): assert result.exit_code == 0 assert "list" in result.output.lower() + def test_watch_command_registered(self, cli_runner): + """Should have watch command registered (backlog task 013)""" + result = cli_runner.invoke(cli, ["watch", "--help"]) + + assert result.exit_code == 0 + assert "list" in result.output.lower() + assert "sync" in result.output.lower() + + def test_watch_has_no_add_subcommand(self, cli_runner): + """ + Invariant 6: the system never authors its own watches. + + Asserted here as well as in tests/cli/test_watch_cli.py because this is + the file someone reads when adding a command. + """ + assert "add" not in cli.commands["watch"].commands + @pytest.mark.parametrize( "gone", ["brief", "frames", "diet", "questions", "predictions", "forecast", "decisions", "beat"], diff --git a/tests/cli/test_watch_cli.py b/tests/cli/test_watch_cli.py new file mode 100644 index 0000000..e6f3157 --- /dev/null +++ b/tests/cli/test_watch_cli.py @@ -0,0 +1,178 @@ +""" +Tests for the `watch` CLI command (backlog task 013). + +:class:`TestNoWriteSeamExists` is the invariant-6 test. It asserts on the +registered command table and on each command's parameters rather than on +anybody's intention, because the way the system starts authoring its own +watches is not a decision -- it is a convenient flag added by someone who has +not read the invariant. +""" + +from contextlib import contextmanager +from datetime import date +from pathlib import Path +from unittest.mock import patch + +import click +import pytest + +from src.cli.watch import _expiry_phrase, watch_command +from src.database.models import Watch as WatchRow + +EXAMPLE_POSITION = Path(__file__).resolve().parents[2] / "config" / "position.example.yaml" +EXAMPLE_WATCHES = Path(__file__).resolve().parents[2] / "config" / "watches.example.yaml" + + +def _patch_db(session): + @contextmanager + def _ctx(): + try: + yield session + session.commit() + except Exception: + session.rollback() + raise + + return patch("src.cli.watch.get_db", _ctx) + + +@pytest.fixture +def stored_watch(test_session): + test_session.add( + WatchRow( + id="conmon-scope-expands", + claim="Continuous monitoring scope expands.", + belief=0.35, + decision_key="fedramp-authorization-renewal", + so_what="It moves renewal past the point where lapsing is cheaper.", + triggers=[{"terms": ["ConMon"], "entities": ["FedRAMP PMO"]}], + expires=date(2027, 3, 31), + staleness_alert_days=30, + ) + ) + test_session.commit() + return test_session + + +class TestWatchList: + def test_empty_table_says_so(self, cli_runner, test_session): + with _patch_db(test_session): + result = cli_runner.invoke(watch_command, ["list"]) + + assert result.exit_code == 0 + assert "No watches stored" in result.output + + def test_shows_belief_decision_and_days_to_expiry(self, cli_runner, stored_watch, monkeypatch): + monkeypatch.setenv("POSITION_PATH", str(EXAMPLE_POSITION)) + with ( + _patch_db(stored_watch), + patch("src.cli.watch.load_position") as loader, + ): + from src.position import load_position as real_loader + + loader.side_effect = lambda: real_loader(EXAMPLE_POSITION) + result = cli_runner.invoke(watch_command, ["list"]) + + assert result.exit_code == 0 + assert "conmon-scope-expands" in result.output + assert "0.35" in result.output + assert "fedramp-authorization-renewal" in result.output + assert "Renew the platform's FedRAMP" in result.output + assert "to expiry" in result.output + + def test_degrades_to_keys_when_position_is_absent(self, cli_runner, stored_watch, tmp_path): + """ + Position lives in a private repo and may not be on this machine. + + The listing then shows decision keys and says why, rather than failing: + the keys are stored on the row and are the load-bearing half. + """ + with ( + _patch_db(stored_watch), + patch("src.cli.watch.load_position", side_effect=FileNotFoundError("no position")), + ): + result = cli_runner.invoke(watch_command, ["list"]) + + assert result.exit_code == 0 + assert "fedramp-authorization-renewal" in result.output + assert "Position unreadable" in result.output + + +class TestWatchSync: + def test_sync_loads_the_checked_in_files(self, cli_runner, test_session, monkeypatch): + monkeypatch.setenv("POSITION_PATH", str(EXAMPLE_POSITION)) + monkeypatch.setenv("WATCHES_PATH", str(EXAMPLE_WATCHES)) + + from src.position import load_position as real_position + from src.position import load_watches as real_watches + + today = date(2026, 9, 1) + with ( + _patch_db(test_session), + patch( + "src.cli.watch.load_position", + side_effect=lambda **kw: real_position(EXAMPLE_POSITION, today=today), + ), + patch( + "src.cli.watch.load_watches", + side_effect=lambda **kw: real_watches( + EXAMPLE_WATCHES, position=kw["position"], today=today + ), + ), + ): + result = cli_runner.invoke(watch_command, ["sync"]) + + assert result.exit_code == 0, result.output + assert test_session.query(WatchRow).count() == 3 + assert "added" in result.output + + def test_sync_reports_a_bad_file_and_stores_nothing(self, cli_runner, test_session): + from src.position import WatchError + + with ( + _patch_db(test_session), + patch("src.cli.watch.load_position"), + patch( + "src.cli.watch.load_watches", + side_effect=WatchError(Path("watches.yaml"), ["watch 'x': 'so_what' is required"]), + ), + ): + result = cli_runner.invoke(watch_command, ["sync"]) + + assert result.exit_code != 0 + assert "so_what" in result.output + assert test_session.query(WatchRow).count() == 0 + + +class TestNoWriteSeamExists: + """Invariant 6: the system never authors its own watches.""" + + def test_only_list_and_sync_are_registered(self): + assert set(watch_command.commands) == {"list", "sync"} + + @pytest.mark.parametrize("forbidden", ["add", "create", "new", "propose", "accept", "edit"]) + def test_no_write_command_exists(self, forbidden): + assert forbidden not in watch_command.commands + + def test_sync_takes_no_arguments_describing_a_watch(self): + """ + ``watch sync`` mirrors a file. The moment it takes ``--claim`` it is a + constructor, and a constructor can be called by a model. + """ + params = watch_command.commands["sync"].params + assert [p.name for p in params if not isinstance(p, click.Option)] == [] + assert params == [] + + def test_list_is_read_only_in_its_signature(self): + assert watch_command.commands["list"].params == [] + + +class TestExpiryPhrase: + def test_future(self): + assert "12d to expiry" in _expiry_phrase(date(2026, 9, 13), date(2026, 9, 1)) + + def test_today(self): + assert "expires today" in _expiry_phrase(date(2026, 9, 1), date(2026, 9, 1)) + + def test_past(self): + assert "expired 5d ago" in _expiry_phrase(date(2026, 8, 27), date(2026, 9, 1)) diff --git a/tests/database/test_add_watches_table.py b/tests/database/test_add_watches_table.py new file mode 100644 index 0000000..1c5bded --- /dev/null +++ b/tests/database/test_add_watches_table.py @@ -0,0 +1,89 @@ +""" +Tests for the migration that adds the watches table (backlog task 013). + +The migration builds the table from the model, so the tests are weighted +towards the properties that would be silently lost if someone later replaced it +with hand-written DDL: that the CHECK constraints exist in the created table, +and that the way down -- which does lose rows -- refuses without the flag. + +Every fixture builds its own SQLite file. Nothing here opens the real database. +""" + +import pytest +from sqlalchemy import create_engine, inspect, text + +from src.database.migrations import add_watches_table as mig +from src.database.models import Watch + + +@pytest.fixture +def blank_engine(tmp_path): + return create_engine(f"sqlite:///{tmp_path / 'fresh.db'}") + + +class TestUpgrade: + def test_creates_the_table(self, blank_engine): + assert mig.upgrade(blank_engine) is True + assert inspect(blank_engine).has_table("watches") + + def test_is_idempotent(self, blank_engine): + mig.upgrade(blank_engine) + assert mig.upgrade(blank_engine) is False + + def test_columns_match_the_model(self, blank_engine): + mig.upgrade(blank_engine) + created = {c["name"] for c in inspect(blank_engine).get_columns("watches")} + assert created == set(Watch.__table__.columns.keys()) + + def test_carries_the_seven_fields_the_spec_names(self, blank_engine): + mig.upgrade(blank_engine) + created = {c["name"] for c in inspect(blank_engine).get_columns("watches")} + assert { + "id", + "claim", + "belief", + "so_what", + "triggers", + "expires", + "staleness_alert_days", + } <= created + + def test_check_constraints_are_in_the_created_ddl(self, blank_engine): + """ + Invariant 2 is enforced at the storage layer, so the DDL has to carry it. + + Read from sqlite_master rather than from the model: the question is what + the migration actually created, not what it intended to. + """ + mig.upgrade(blank_engine) + with blank_engine.connect() as conn: + ddl = conn.execute( + text("SELECT sql FROM sqlite_master WHERE type='table' AND name='watches'") + ).scalar() + + assert "ck_watches_so_what_present" in ddl + assert "ck_watches_decision_present" in ddl + assert "ck_watches_belief_range" in ddl + assert "ck_watches_staleness_min" in ddl + + def test_touches_nothing_else(self, blank_engine): + """Additive means additive: it creates one table and no others.""" + mig.upgrade(blank_engine) + assert inspect(blank_engine).get_table_names() == ["watches"] + + +class TestDowngrade: + def test_refuses_without_confirm(self, blank_engine): + mig.upgrade(blank_engine) + with pytest.raises(SystemExit) as exc: + mig.downgrade(blank_engine) + assert "--confirm" in str(exc.value) + assert inspect(blank_engine).has_table("watches") + + def test_drops_with_confirm(self, blank_engine): + mig.upgrade(blank_engine) + assert mig.downgrade(blank_engine, confirmed=True) is True + assert not inspect(blank_engine).has_table("watches") + + def test_drop_on_a_missing_table_is_a_no_op(self, blank_engine): + assert mig.downgrade(blank_engine, confirmed=True) is False diff --git a/tests/database/test_connection.py b/tests/database/test_connection.py index 6b5d12b..a7fbe36 100644 --- a/tests/database/test_connection.py +++ b/tests/database/test_connection.py @@ -116,10 +116,11 @@ def test_create_tables_creates_all_tables(self, tmp_path, mocker): inspector = inspect(engine) table_names = inspector.get_table_names() - # Exactly two, asserted as a set rather than a subset: backlog task 012 - # deleted the other nineteen models, and a subset check would pass again - # the moment one of them was reintroduced by accident. - assert set(table_names) == {"rss_feeds", "articles"} + # Exactly three, asserted as a set rather than a subset: backlog task + # 012 deleted nineteen models, and a subset check would pass again the + # moment one of them was reintroduced by accident. `watches` was added + # deliberately by task 013 (2026-08-31) and is the only addition since. + assert set(table_names) == {"rss_feeds", "articles", "watches"} def test_create_tables_is_idempotent(self, tmp_path, mocker): """create_tables should be safe to call multiple times""" diff --git a/tests/database/test_models.py b/tests/database/test_models.py index a77b756..15edfd5 100644 --- a/tests/database/test_models.py +++ b/tests/database/test_models.py @@ -184,12 +184,15 @@ def test_no_table_in_the_schema_is_person_shaped(self, test_engine): assert not [name for name in names if any(word in name.lower() for word in forbidden)] - def test_only_the_two_surviving_tables_are_mapped(self): + def test_only_the_declared_tables_are_mapped(self): """ Task 012 removed nineteen models. If one comes back by accident, ``create_tables()`` would recreate a concept the rewrite deleted, so the mapped set is pinned rather than left to review. + + ``watches`` joined the set on 2026-08-31 (backlog task 013). Every + addition to this set should be a task that says so. """ from src.database.models import Base - assert set(Base.metadata.tables) == {"rss_feeds", "articles"} + assert set(Base.metadata.tables) == {"rss_feeds", "articles", "watches"} diff --git a/tests/position/__init__.py b/tests/position/__init__.py new file mode 100644 index 0000000..e69de29 diff --git a/tests/position/conftest.py b/tests/position/conftest.py new file mode 100644 index 0000000..9b76890 --- /dev/null +++ b/tests/position/conftest.py @@ -0,0 +1,118 @@ +""" +Fixtures for the Position and Watch loaders. + +Every test writes its own YAML into ``tmp_path``. Nothing here reads the +operator's real Position -- it is not in this repository and must not be +required to run the suite. + +``TODAY`` is frozen rather than ``date.today()``. The expiry rules are +date-relative, so a suite that used the wall clock would start failing on the +day the fixture dates went past, which is the kind of test failure that teaches +nothing. +""" + +from datetime import date +from pathlib import Path + +import pytest +import yaml + +from src.position import load_position + +TODAY = date(2026, 9, 1) + + +def _position_doc() -> dict: + return { + "version": 1, + "reviewed": date(2026, 8, 31), + "decisions": [ + { + "key": "renew-authorization", + "name": "Renew the authorization or let it lapse", + "deadline": date(2027, 3, 31), + "stake": "Two engineer-quarters against 40% of federal revenue.", + }, + { + "key": "hire-engineer", + "name": "Hire a compliance engineer", + "deadline": date(2026, 12, 15), + "stake": "A missed deadline nobody owns.", + }, + ], + } + + +def _watch_doc(**overrides) -> dict: + """One valid watch, with any field replaced. ``None`` removes the field.""" + watch = { + "id": "conmon-scope-expands", + "claim": "Continuous monitoring scope expands before the assessment window.", + "belief": 0.35, + "so_what": { + "decision": "renew-authorization", + "because": "Scope expansion moves renewal past the point where lapsing is cheaper.", + }, + "triggers": [ + {"entities": ["FedRAMP PMO"], "terms": ["continuous monitoring", "ConMon"]}, + {"sources": ["Federal Register"], "terms": ["FedRAMP"]}, + ], + "expires": date(2027, 3, 31), + "staleness_alert_days": 30, + } + for key, value in overrides.items(): + if value is _ABSENT: + watch.pop(key, None) + else: + watch[key] = value + return watch + + +class _Absent: + def __repr__(self) -> str: # pragma: no cover - debugging aid + return "" + + +_ABSENT = _Absent() +ABSENT = _ABSENT + + +@pytest.fixture +def write_yaml(tmp_path): + """Write a document (or raw text) to ``tmp_path`` and hand back the path.""" + + def _write(name: str, document) -> Path: + target = tmp_path / name + if isinstance(document, str): + target.write_text(document, encoding="utf-8") + else: + target.write_text(yaml.safe_dump(document, sort_keys=False), encoding="utf-8") + return target + + return _write + + +@pytest.fixture +def position_doc(): + return _position_doc() + + +@pytest.fixture +def watch_doc(): + return _watch_doc + + +@pytest.fixture +def position(write_yaml, position_doc): + """A loaded, valid Position with two decisions.""" + return load_position(write_yaml("position.yaml", position_doc), today=TODAY) + + +@pytest.fixture +def watches_file(write_yaml): + """Writes a watches file from a list of watch dicts.""" + + def _write(*watches) -> Path: + return write_yaml("watches.yaml", {"version": 1, "watches": list(watches)}) + + return _write diff --git a/tests/position/test_position.py b/tests/position/test_position.py new file mode 100644 index 0000000..fcd6260 --- /dev/null +++ b/tests/position/test_position.py @@ -0,0 +1,196 @@ +""" +Tests for the Position loader (backlog task 013). + +The split these tests are written against: structural problems are rejections, +judgement calls are warnings. A decision with no deadline cannot be repaired by +the loader and is refused; a Position that has grown to three pages might be +exactly right and is only named. +""" + +from datetime import date +from pathlib import Path + +import pytest + +from src.position import MAX_PAGES, POSITION_PAGE_WORDS, Position, PositionError, load_position + +from .conftest import TODAY + +EXAMPLE = Path(__file__).resolve().parents[2] / "config" / "position.example.yaml" + + +class TestLoadsValidPosition: + def test_loads_decisions(self, position): + assert isinstance(position, Position) + assert position.decision_keys == {"renew-authorization", "hire-engineer"} + + def test_decision_lookup(self, position): + found = position.decision("hire-engineer") + assert found is not None + assert found.deadline == date(2026, 12, 15) + assert found.days_remaining(TODAY) == 105 + + def test_unknown_decision_lookup_returns_none(self, position): + assert position.decision("not-a-decision") is None + + def test_reviewed_date(self, position): + assert position.reviewed == date(2026, 8, 31) + + def test_valid_position_warns_about_nothing(self, position): + assert position.warnings == () + + +class TestCheckedInExample: + """ + The example in this repo has to actually load. + + An example file that does not parse is worse than no example: it is the + first thing a new operator copies. + """ + + def test_example_loads(self): + loaded = load_position(EXAMPLE, today=TODAY) + assert len(loaded.decisions) == 3 + assert all(d.deadline > TODAY for d in loaded.decisions) + + def test_example_is_under_two_pages(self): + loaded = load_position(EXAMPLE, today=TODAY) + assert loaded.word_count <= POSITION_PAGE_WORDS * MAX_PAGES + assert loaded.warnings == () + + +class TestMissingFile: + def test_absent_position_fails_fast(self, tmp_path): + with pytest.raises(FileNotFoundError) as exc: + load_position(tmp_path / "nope.yaml") + assert "POSITION_PATH" in str(exc.value) + + def test_error_points_at_the_example(self, tmp_path): + with pytest.raises(FileNotFoundError) as exc: + load_position(tmp_path / "nope.yaml") + assert "position.example.yaml" in str(exc.value) + + +class TestRejections: + def test_decision_without_deadline_is_rejected(self, write_yaml, position_doc): + del position_doc["decisions"][0]["deadline"] + path = write_yaml("position.yaml", position_doc) + with pytest.raises(PositionError) as exc: + load_position(path, today=TODAY) + assert "'deadline' is required" in str(exc.value) + + def test_deadline_that_is_not_a_date_is_rejected(self, write_yaml, position_doc): + position_doc["decisions"][0]["deadline"] = "when the budget lands" + path = write_yaml("position.yaml", position_doc) + with pytest.raises(PositionError, match="is not a date"): + load_position(path, today=TODAY) + + def test_missing_name_is_rejected(self, write_yaml, position_doc): + position_doc["decisions"][1]["name"] = " " + path = write_yaml("position.yaml", position_doc) + with pytest.raises(PositionError, match="'name' is required"): + load_position(path, today=TODAY) + + def test_duplicate_keys_are_rejected(self, write_yaml, position_doc): + position_doc["decisions"][1]["key"] = "renew-authorization" + path = write_yaml("position.yaml", position_doc) + with pytest.raises(PositionError, match="duplicate decision key"): + load_position(path, today=TODAY) + + def test_non_slug_key_is_rejected(self, write_yaml, position_doc): + position_doc["decisions"][0]["key"] = "Renew The Authorization" + path = write_yaml("position.yaml", position_doc) + with pytest.raises(PositionError, match="is not a slug"): + load_position(path, today=TODAY) + + def test_empty_decisions_is_rejected(self, write_yaml, position_doc): + position_doc["decisions"] = [] + path = write_yaml("position.yaml", position_doc) + with pytest.raises(PositionError, match="holds no stakes"): + load_position(path, today=TODAY) + + def test_unknown_field_is_rejected(self, write_yaml, position_doc): + position_doc["decisions"][0]["urgency"] = "high" + path = write_yaml("position.yaml", position_doc) + with pytest.raises(PositionError, match="unknown field"): + load_position(path, today=TODAY) + + def test_empty_file_is_rejected(self, write_yaml): + path = write_yaml("position.yaml", "# nothing but a comment\n") + with pytest.raises(PositionError, match="empty"): + load_position(path, today=TODAY) + + def test_malformed_yaml_is_rejected(self, write_yaml): + path = write_yaml("position.yaml", "decisions: [\n - key: x\n") + with pytest.raises(PositionError, match="not valid YAML"): + load_position(path, today=TODAY) + + def test_every_problem_is_reported_at_once(self, write_yaml, position_doc): + del position_doc["decisions"][0]["deadline"] + position_doc["decisions"][1]["name"] = "" + path = write_yaml("position.yaml", position_doc) + with pytest.raises(PositionError) as exc: + load_position(path, today=TODAY) + assert len(exc.value.problems) == 2 + + +class TestWarnsRatherThanFails: + def test_long_position_warns_and_still_loads(self, write_yaml, position_doc): + """ + Over two pages the loader names the drift. It does not refuse. + + A three-page Position of genuine deadlines beats a one-page Position of + vague interests, so length can only ever be a flag. + """ + filler = " ".join(["exposure"] * (POSITION_PAGE_WORDS * MAX_PAGES + 50)) + position_doc["decisions"][0]["stake"] = filler + path = write_yaml("position.yaml", position_doc) + + loaded = load_position(path, today=TODAY) + + assert loaded.word_count > POSITION_PAGE_WORDS * MAX_PAGES + assert len(loaded.decisions) == 2 + assert any("interests" in note for note in loaded.warnings) + assert any("pages" in note for note in loaded.warnings) + + def test_warning_is_logged_loudly(self, write_yaml, position_doc, caplog): + filler = " ".join(["exposure"] * (POSITION_PAGE_WORDS * MAX_PAGES + 50)) + position_doc["decisions"][0]["stake"] = filler + path = write_yaml("position.yaml", position_doc) + + with caplog.at_level("WARNING", logger="src.position.position"): + load_position(path, today=TODAY) + + assert any("interests" in record.message for record in caplog.records) + + def test_comments_do_not_count_towards_the_page_limit(self, write_yaml, position_doc): + """ + The operator's notes are not drift. + + Counting comments would make the warning fire on a well-annotated file, + which is the opposite of what it is for. + """ + comment = "\n".join(["# " + " ".join(["note"] * 20)] * 60) + path = write_yaml("position.yaml", position_doc) + path.write_text(comment + "\n" + path.read_text(encoding="utf-8"), encoding="utf-8") + + loaded = load_position(path, today=TODAY) + + assert loaded.warnings == () + + def test_past_deadline_warns(self, write_yaml, position_doc): + position_doc["decisions"][0]["deadline"] = date(2026, 1, 1) + path = write_yaml("position.yaml", position_doc) + + loaded = load_position(path, today=TODAY) + + assert len(loaded.decisions) == 2 + assert any("passed its deadline" in note for note in loaded.warnings) + + def test_missing_stake_warns(self, write_yaml, position_doc): + del position_doc["decisions"][0]["stake"] + path = write_yaml("position.yaml", position_doc) + + loaded = load_position(path, today=TODAY) + + assert any("states no 'stake'" in note for note in loaded.warnings) diff --git a/tests/position/test_watches.py b/tests/position/test_watches.py new file mode 100644 index 0000000..a4f4f18 --- /dev/null +++ b/tests/position/test_watches.py @@ -0,0 +1,337 @@ +""" +Tests for the Watch loader and the watches table (backlog task 013). + +The class that matters most here is :class:`TestNothingInvalidIsEverStored`. The +acceptance criterion is not "an invalid watch raises" -- it is that an invalid +watch is *not stored*, which is a claim about the database and has to be checked +against the database. Each case asserts the row count is unchanged across the +rejection, and the class ends by accepting a valid watch so that a suite which +passed because the write path was broken would fail. + +Invariant 2 -- every Watch names a decision -- is enforced twice on purpose: +once by the loader and once by a CHECK constraint, tested separately below with +raw SQL that bypasses the loader entirely. +""" + +from datetime import date +from pathlib import Path + +import pytest +from sqlalchemy import text +from sqlalchemy.exc import IntegrityError + +from src.database.models import Watch as WatchRow +from src.position import TriggerClause, WatchError, load_position, load_watches, sync_watches + +from .conftest import ABSENT, TODAY + +EXAMPLE_POSITION = Path(__file__).resolve().parents[2] / "config" / "position.example.yaml" +EXAMPLE_WATCHES = Path(__file__).resolve().parents[2] / "config" / "watches.example.yaml" + + +def _load(path, position): + return load_watches(path, position=position, today=TODAY) + + +class TestLoadsValidWatches: + def test_loads_one_watch(self, watches_file, watch_doc, position): + (watch,) = _load(watches_file(watch_doc()), position) + + assert watch.id == "conmon-scope-expands" + assert watch.belief == pytest.approx(0.35) + assert watch.decision_key == "renew-authorization" + assert watch.so_what.startswith("Scope expansion") + assert watch.expires == date(2027, 3, 31) + assert watch.staleness_alert_days == 30 + + def test_days_to_expiry(self, watches_file, watch_doc, position): + (watch,) = _load(watches_file(watch_doc()), position) + assert watch.days_to_expiry(TODAY) == 211 + + def test_triggers_are_structured_clauses(self, watches_file, watch_doc, position): + (watch,) = _load(watches_file(watch_doc()), position) + + assert all(isinstance(clause, TriggerClause) for clause in watch.triggers) + assert watch.triggers[0].entities == ("FedRAMP PMO",) + assert watch.triggers[0].terms == ("continuous monitoring", "ConMon") + assert watch.triggers[1].sources == ("Federal Register",) + + def test_triggers_json_omits_empty_fields(self, watches_file, watch_doc, position): + (watch,) = _load(watches_file(watch_doc()), position) + + assert watch.triggers_json() == [ + {"terms": ["continuous monitoring", "ConMon"], "entities": ["FedRAMP PMO"]}, + {"terms": ["FedRAMP"], "sources": ["Federal Register"]}, + ] + + def test_staleness_accepts_a_cadence_string(self, watches_file, watch_doc, position): + """ + ``2w`` goes through src/utils/cadence.py rather than a second parser. + + One interval grammar across the system; the cadence module survived task + 012 for exactly this. + """ + (watch,) = _load(watches_file(watch_doc(staleness_alert_days="2w")), position) + assert watch.staleness_alert_days == 14 + + +class TestCheckedInExample: + def test_example_watches_load_against_example_position(self): + position = load_position(EXAMPLE_POSITION, today=TODAY) + watches = load_watches(EXAMPLE_WATCHES, position=position, today=TODAY) + + assert len(watches) == 3 + assert {w.decision_key for w in watches} <= position.decision_keys + assert all(w.triggers for w in watches) + + def test_example_watches_all_name_a_real_decision(self): + position = load_position(EXAMPLE_POSITION, today=TODAY) + watches = load_watches(EXAMPLE_WATCHES, position=position, today=TODAY) + + for watch in watches: + assert position.decision(watch.decision_key) is not None + + +class TestMissingFile: + def test_absent_watches_file_fails_fast(self, tmp_path, position): + with pytest.raises(FileNotFoundError) as exc: + load_watches(tmp_path / "nope.yaml", position=position, today=TODAY) + assert "WATCHES_PATH" in str(exc.value) + + +# Every rejection the acceptance criteria name, as (label, watch overrides). +# ``ABSENT`` removes the field entirely. +REJECTIONS = [ + ("so_what empty string", {"so_what": ""}, "is empty"), + ("so_what whitespace only", {"so_what": " \t "}, "is empty"), + ("so_what absent", {"so_what": ABSENT}, "'so_what' is required"), + ("so_what is bare prose", {"so_what": "it matters for the renewal"}, "must be a mapping"), + ( + "so_what.because blank", + {"so_what": {"decision": "renew-authorization", "because": " "}}, + "because is required", + ), + ( + "so_what.decision blank", + {"so_what": {"decision": "", "because": "it moves the renewal"}}, + "decision is required", + ), + ( + "so_what names no decision in the Position", + {"so_what": {"decision": "buy-a-boat", "because": "it moves the renewal"}}, + "names no decision", + ), + ("belief above 1.0", {"belief": 1.4}, "outside [0.0, 1.0]"), + ("belief below 0.0", {"belief": -0.1}, "outside [0.0, 1.0]"), + ("belief is prose", {"belief": "fairly likely"}, "must be a number"), + ("belief absent", {"belief": ABSENT}, "'belief' is required"), + ("expires in the past", {"expires": date(2026, 1, 1)}, "is in the past"), + ("expires absent", {"expires": ABSENT}, "'expires' is required"), + ("expires is not a date", {"expires": "soon"}, "is not a date"), + ("staleness_alert_days zero", {"staleness_alert_days": 0}, "at least 1"), + ("staleness_alert_days negative", {"staleness_alert_days": -5}, "at least 1"), + ("staleness_alert_days absent", {"staleness_alert_days": ABSENT}, "is required"), + ( + "staleness_alert_days is prose", + {"staleness_alert_days": "when it goes quiet"}, + "not an interval", + ), + ("claim blank", {"claim": " "}, "'claim' is required"), + ("id absent", {"id": ABSENT}, "'id' is required"), + ("id is not a slug", {"id": "Watch The Renewal"}, "is not a slug"), + ("triggers absent", {"triggers": ABSENT}, "'triggers' is required"), + ("triggers empty", {"triggers": []}, "never fires"), + ("triggers is prose", {"triggers": "anything about FedRAMP"}, "is prose"), + ( + "trigger clause is a sentence", + {"triggers": ["watch for anything that changes the scope"]}, + "not the sentence", + ), + ("trigger clause constrains nothing", {"triggers": [{}]}, "constrains nothing"), + ( + "trigger field is prose", + {"triggers": [{"terms": "continuous monitoring"}]}, + "must be a list of terms", + ), + ( + "trigger clause has an unknown field", + {"triggers": [{"terms": ["FedRAMP"], "vibe": ["ominous"]}]}, + "unknown field", + ), + ("unknown watch field", {"urgency": "high"}, "unknown field"), +] + + +class TestNothingInvalidIsEverStored: + """ + Rejection is a claim about the table, so it is checked against the table. + + A loader that raised and then wrote anyway would pass a `pytest.raises` + suite. The row count is what makes these bite. + """ + + @pytest.mark.parametrize( + ("label", "overrides", "expected"), + REJECTIONS, + ids=[case[0] for case in REJECTIONS], + ) + def test_rejected_and_not_stored( + self, label, overrides, expected, watches_file, watch_doc, position, test_session + ): + before = test_session.query(WatchRow).count() + path = watches_file(watch_doc(**overrides)) + + # Deliberately not `pytest.raises`. The claim under test is about the + # table, so the row count has to be asserted whether or not the loader + # raised -- a loader that quietly accepted this watch must fail on the + # count, not on a missing exception, so the failure names the real + # problem. + raised = None + try: + sync_watches(test_session, _load(path, position)) + test_session.commit() + except WatchError as exc: + raised = exc + + assert test_session.query(WatchRow).count() == before, f"{label}: was stored" + assert raised is not None, f"{label}: accepted a watch that must be rejected" + assert expected in str(raised), f"{label}: unhelpful error {raised}" + + def test_a_valid_watch_is_accepted(self, watches_file, watch_doc, position, test_session): + """ + The control. Without it every case above could pass on a broken writer. + """ + assert test_session.query(WatchRow).count() == 0 + + watches = _load(watches_file(watch_doc()), position) + sync_watches(test_session, watches) + test_session.commit() + + assert test_session.query(WatchRow).count() == 1 + row = test_session.query(WatchRow).one() + assert row.decision_key == "renew-authorization" + assert row.triggers[0]["terms"] == ["continuous monitoring", "ConMon"] + + def test_one_bad_watch_stores_none_of_the_others( + self, watches_file, watch_doc, position, test_session + ): + """ + A file is applied whole or not at all. + + Storing the good half leaves the table in a state that matches no file, + which is the state nobody can debug six months later. + """ + good = watch_doc(id="good-one") + bad = watch_doc(id="bad-one", so_what={"decision": "buy-a-boat", "because": "no"}) + + with pytest.raises(WatchError): + watches = _load(watches_file(good, bad), position) + sync_watches(test_session, watches) + + assert test_session.query(WatchRow).count() == 0 + + +class TestOtherRejections: + def test_duplicate_ids_are_rejected(self, watches_file, watch_doc, position): + with pytest.raises(WatchError, match="duplicate watch id"): + _load(watches_file(watch_doc(), watch_doc()), position) + + def test_empty_file_is_rejected(self, write_yaml, position): + with pytest.raises(WatchError, match="empty"): + _load(write_yaml("watches.yaml", "# nothing here\n"), position) + + def test_malformed_yaml_is_rejected(self, write_yaml, position): + with pytest.raises(WatchError, match="not valid YAML"): + _load(write_yaml("watches.yaml", "watches: [\n - id: x\n"), position) + + def test_every_problem_is_reported_at_once(self, watches_file, watch_doc, position): + path = watches_file(watch_doc(belief=3.0, so_what=ABSENT, staleness_alert_days=0)) + with pytest.raises(WatchError) as exc: + _load(path, position) + assert len(exc.value.problems) == 3 + + def test_error_names_the_known_decision_keys(self, watches_file, watch_doc, position): + path = watches_file(watch_doc(so_what={"decision": "buy-a-boat", "because": "no"})) + with pytest.raises(WatchError) as exc: + _load(path, position) + assert "renew-authorization" in str(exc.value) + + +class TestSync: + def test_sync_adds_updates_and_removes(self, watches_file, watch_doc, position, test_session): + first = _load(watches_file(watch_doc(id="a"), watch_doc(id="b")), position) + summary = sync_watches(test_session, first) + test_session.commit() + assert sorted(summary["added"]) == ["a", "b"] + + # The file is authoritative: "b" is gone and "a" moved. + second = _load(watches_file(watch_doc(id="a", belief=0.9)), position) + summary = sync_watches(test_session, second) + test_session.commit() + + assert summary["updated"] == ["a"] + assert summary["removed"] == ["b"] + assert test_session.query(WatchRow).count() == 1 + assert test_session.query(WatchRow).one().belief == pytest.approx(0.9) + + +class TestSchemaEnforcesInvariantTwo: + """ + The constraints, exercised with raw SQL so the loader is out of the picture. + + Invariant 2 is enforced in the schema, which means a row inserted by hand + through sqlite3 has to fail the same way a bad YAML file does. This is the + half of the enforcement the loader tests cannot reach. + """ + + def _insert(self, session, **overrides): + row = { + "id": "x", + "claim": "a claim", + "belief": 0.5, + "decision_key": "renew-authorization", + "so_what": "it moves the renewal", + "triggers": "[]", + "expires": "2027-03-31", + "staleness_alert_days": 30, + } + row.update(overrides) + session.execute( + text( + "INSERT INTO watches (id, claim, belief, decision_key, so_what, triggers," + " expires, staleness_alert_days) VALUES (:id, :claim, :belief, :decision_key," + " :so_what, :triggers, :expires, :staleness_alert_days)" + ), + row, + ) + + def test_blank_so_what_is_refused_by_the_database(self, test_session): + with pytest.raises(IntegrityError): + self._insert(test_session, so_what=" ") + test_session.rollback() + assert test_session.query(WatchRow).count() == 0 + + def test_null_so_what_is_refused_by_the_database(self, test_session): + with pytest.raises(IntegrityError): + self._insert(test_session, so_what=None) + test_session.rollback() + + def test_blank_decision_key_is_refused_by_the_database(self, test_session): + with pytest.raises(IntegrityError): + self._insert(test_session, decision_key="") + test_session.rollback() + + def test_out_of_range_belief_is_refused_by_the_database(self, test_session): + with pytest.raises(IntegrityError): + self._insert(test_session, belief=1.5) + test_session.rollback() + + def test_zero_staleness_is_refused_by_the_database(self, test_session): + with pytest.raises(IntegrityError): + self._insert(test_session, staleness_alert_days=0) + test_session.rollback() + + def test_a_good_row_still_inserts(self, test_session): + self._insert(test_session) + test_session.commit() + assert test_session.query(WatchRow).count() == 1 From 644b442e4d555a176f6df093b1fda2a9fed45bce Mon Sep 17 00:00:00 2001 From: sam-aydlette Date: Mon, 31 Aug 2026 11:51:44 -0400 Subject: [PATCH 2/3] position: split watch validation into its own module watches.py had grown to 583 lines, past the 200-300 the repository asks for. The split is along a real seam rather than a line count: _validate.py takes a raw YAML value and either appends a sentence to `problems` or returns a clean one, and touches neither the file system nor the database; watches.py owns the loaded object, the file, and the sync. validate_watch returns constructor keywords rather than a Watch. That keeps the dependency one-way -- watches -> _validate -> position -- where returning the type would have required _validate to import the module that imports it. It also puts construction in the loader, which is the thing that admits a watch; validation only decides whether it is admissible. Pure move. 554 tests still pass, and the mutation checks still bite: removing any one enforcement (so_what defaulting, an unknown decision tolerated, the belief range, the past-expiry check, the staleness floor, prose triggers, or the all-or-nothing raise) still fails a named test. Co-Authored-By: Claude Opus 5 (1M context) Claude-Session: https://claude.ai/code/session_01M9gY2miAUXijbFm25YnFcq --- src/position/_validate.py | 385 ++++++++++++++++++++++++++++++++++++++ src/position/watches.py | 365 +----------------------------------- 2 files changed, 391 insertions(+), 359 deletions(-) create mode 100644 src/position/_validate.py diff --git a/src/position/_validate.py b/src/position/_validate.py new file mode 100644 index 0000000..7eb76a3 --- /dev/null +++ b/src/position/_validate.py @@ -0,0 +1,385 @@ +""" +Field-by-field validation for a hand-authored watch set. + +Split out of :mod:`src.position.watches` on 2026-08-31 to keep both files +inside the repository's 200-300 line rule. The split is along a real seam: +everything here takes a raw YAML value and either appends a sentence to +``problems`` or returns a clean value, and nothing here touches the database or +the file system. + +**Every function in this module rejects rather than defaults.** That is the +whole point of the module. The deleted prediction ledger accumulated 25 +unfalsifiable rows because a missing field was tolerated at every layer that +could have refused it, and the fix is not better review -- it is a validator +that has no branch in which a blank field survives. + +Problems accumulate rather than raising, so a hand-edited file reports all of +its errors in one run. The caller raises once, before anything is written. +""" + +from __future__ import annotations + +import re +from dataclasses import dataclass +from datetime import date +from typing import Any + +from src.utils.cadence import InvalidCadence, parse_cadence + +from .position import Position, _as_date, _is_blank + +__all__ = [ + "SO_WHAT_KEYS", + "TRIGGER_FIELDS", + "WATCH_KEYS", + "TriggerClause", + "validate_watch", +] + +_ID_RE = re.compile(r"^[a-z0-9][a-z0-9._-]*$") + +WATCH_KEYS = { + "id", + "claim", + "belief", + "so_what", + "triggers", + "expires", + "staleness_alert_days", +} +SO_WHAT_KEYS = {"decision", "because"} + +# The only three axes a trigger clause may constrain. Each maps onto something +# Tier 1 can evaluate without a model: a word-boundary term match, an entity's +# declared surface forms, and the adapter that produced the observation. +TRIGGER_FIELDS = ("terms", "entities", "sources") + + +@dataclass(frozen=True) +class TriggerClause: + """ + One conjunctive clause of a trigger. + + A clause fires when every field it populates matches: any of its ``terms`` + appears, **and** any of its ``entities`` appears, **and** the observation + came from one of its ``sources``. Fields left out are not constraints. The + watch fires when any clause fires, so ``triggers`` as a whole is a + disjunction of conjunctions -- expressive enough to say "FedRAMP *and* + continuous monitoring, or anything about it in the Federal Register", and + small enough that Tier 1 compiles it to regexes and a set membership test + with no interpreter in between. + """ + + terms: tuple[str, ...] = () + entities: tuple[str, ...] = () + sources: tuple[str, ...] = () + + def as_dict(self) -> dict[str, list[str]]: + """The JSON shape stored in ``watches.triggers``. Empty fields are omitted.""" + return {name: list(getattr(self, name)) for name in TRIGGER_FIELDS if getattr(self, name)} + + +def _validate_so_what( + label: str, raw: Any, position: Position, problems: list[str] +) -> tuple[str, str] | None: + """ + Invariant 2, enforced. Returns ``(decision_key, because)`` or None. + + Every branch here rejects. None of them defaults, substitutes a placeholder, + or falls back to the claim text -- a Watch with no articulable decision is + curiosity, and the point of the invariant is that curiosity does not get + stored and then quietly alert. + """ + if raw is None: + problems.append( + f"{label}: 'so_what' is required and names the decision this watch serves. " + f"A watch with no decision attached is curiosity, not monitoring." + ) + return None + if isinstance(raw, str): + if not raw.strip(): + problems.append( + f"{label}: 'so_what' is empty. It must name a decision from the Position and " + f"say what changes; blank is rejected, never defaulted." + ) + else: + problems.append( + f"{label}: 'so_what' must be a mapping of 'decision' (a key from the Position) " + f"and 'because' (what changes), not a bare string." + ) + return None + if not isinstance(raw, dict): + problems.append(f"{label}: 'so_what' must be a mapping, got {type(raw).__name__}") + return None + + unknown = sorted(set(raw) - SO_WHAT_KEYS) + if unknown: + problems.append( + f"{label}: so_what has unknown field(s) {unknown}; allowed: ['because', 'decision']" + ) + + decision_key = raw.get("decision") + because = raw.get("because") + ok = True + + if _is_blank(decision_key): + problems.append( + f"{label}: so_what.decision is required and must name a decision key from the " + f"Position. Known keys: {sorted(position.decision_keys)}" + ) + ok = False + elif str(decision_key).strip() not in position.decision_keys: + problems.append( + f"{label}: so_what.decision '{str(decision_key).strip()}' names no decision in " + f"{position.path}. Known keys: {sorted(position.decision_keys)}. Add the decision " + f"to the Position, or drop the watch." + ) + ok = False + + if _is_blank(because): + problems.append( + f"{label}: so_what.because is required and must be a non-empty sentence saying what " + f"changes for that decision if the claim resolves." + ) + ok = False + + if not ok: + return None + return str(decision_key).strip(), str(because).strip() + + +def _validate_string_list( + label: str, field_name: str, raw: Any, problems: list[str] +) -> tuple[str, ...]: + if raw is None: + return () + if isinstance(raw, str): + problems.append( + f"{label}: trigger '{field_name}' must be a list of terms, not a string. " + f"A trigger is compiled into a predicate, so it cannot be a sentence." + ) + return () + if not isinstance(raw, list): + problems.append(f"{label}: trigger '{field_name}' must be a list, got {type(raw).__name__}") + return () + values: list[str] = [] + for item in raw: + if _is_blank(item): + problems.append(f"{label}: trigger '{field_name}' contains an empty entry") + continue + values.append(str(item).strip()) + return tuple(values) + + +def _validate_triggers(label: str, raw: Any, problems: list[str]) -> tuple[TriggerClause, ...]: + """ + Structure, not prose. Every rejection here is a rejection of prose. + + ``trigger_condition`` in the deleted ledger accepted free text, and 33 rows + were written against it and graded zero times, because nothing downstream + could evaluate a sentence. Tier 1 compiles these clauses; a clause it cannot + compile is refused at the door rather than stored and skipped. + """ + if raw is None: + problems.append( + f"{label}: 'triggers' is required -- what observation would be candidate evidence?" + ) + return () + if isinstance(raw, str): + problems.append( + f"{label}: 'triggers' is prose. It must be a list of clauses over " + f"{list(TRIGGER_FIELDS)}, because Tier 1 compiles triggers into a deterministic " + f"predicate and cannot compile a sentence." + ) + return () + if not isinstance(raw, list): + problems.append(f"{label}: 'triggers' must be a list, got {type(raw).__name__}") + return () + if not raw: + problems.append(f"{label}: 'triggers' is empty -- a watch nothing can trigger never fires") + return () + + clauses: list[TriggerClause] = [] + for index, entry in enumerate(raw): + clause_label = f"{label}.triggers[{index}]" + if isinstance(entry, str): + problems.append( + f"{clause_label}: a trigger clause is a mapping over {list(TRIGGER_FIELDS)}, " + f"not the sentence {entry.strip()[:60]!r}. Prose triggers are what left 33 " + f"predictions ungradeable; write the terms and entities that would match." + ) + continue + if not isinstance(entry, dict): + problems.append(f"{clause_label}: expected a mapping, got {type(entry).__name__}") + continue + unknown = sorted(set(entry) - set(TRIGGER_FIELDS)) + if unknown: + problems.append( + f"{clause_label}: unknown field(s) {unknown}; a clause may constrain only " + f"{list(TRIGGER_FIELDS)}" + ) + continue + clause = TriggerClause( + terms=_validate_string_list(clause_label, "terms", entry.get("terms"), problems), + entities=_validate_string_list( + clause_label, "entities", entry.get("entities"), problems + ), + sources=_validate_string_list(clause_label, "sources", entry.get("sources"), problems), + ) + if not (clause.terms or clause.entities or clause.sources): + problems.append( + f"{clause_label}: constrains nothing. A clause with no terms, entities or " + f"sources matches every observation." + ) + continue + clauses.append(clause) + return tuple(clauses) + + +def _validate_belief(label: str, raw: Any, problems: list[str]) -> float | None: + if raw is None: + problems.append(f"{label}: 'belief' is required -- a probability in [0.0, 1.0]") + return None + if isinstance(raw, bool) or not isinstance(raw, int | float): + problems.append(f"{label}: belief must be a number in [0.0, 1.0], got {raw!r}") + return None + value = float(raw) + if not (0.0 <= value <= 1.0): + problems.append(f"{label}: belief {value} is outside [0.0, 1.0]") + return None + return value + + +def _validate_staleness(label: str, raw: Any, problems: list[str]) -> int | None: + """ + Days, written as an integer or as a cadence (``2w``, ``30d``). + + The cadence spelling goes through :mod:`src.utils.cadence` rather than a + second parser, so the whole system reads one interval grammar. + """ + if raw is None: + problems.append( + f"{label}: 'staleness_alert_days' is required -- after how much silence is the " + f"silence itself the finding?" + ) + return None + if isinstance(raw, bool): + problems.append( + f"{label}: staleness_alert_days must be a whole number of days, got {raw!r}" + ) + return None + if isinstance(raw, str): + try: + days = parse_cadence(raw).days + except InvalidCadence as exc: + problems.append(f"{label}: staleness_alert_days {raw!r} is not an interval -- {exc}") + return None + elif isinstance(raw, int): + days = raw + else: + problems.append( + f"{label}: staleness_alert_days must be a whole number of days, got {raw!r}" + ) + return None + if days < 1: + problems.append( + f"{label}: staleness_alert_days is {days}; it must be at least 1. Zero would make " + f"every watch stale the moment it is loaded." + ) + return None + return days + + +def validate_watch( + index: int, + raw: Any, + position: Position, + seen: set[str], + today: date, + problems: list[str], +) -> dict[str, Any] | None: + """ + One watch entry, validated. Returns constructor keywords, or None. + + It returns keywords rather than a :class:`~src.position.watches.Watch` + deliberately: the validator would otherwise have to import the type from the + module that imports it, and the loader is the right place for construction + anyway -- validation decides whether a watch is admissible, the loader is + what admits it. + """ + label = f"watches[{index}]" + if not isinstance(raw, dict): + problems.append(f"{label}: expected a mapping, got {type(raw).__name__}") + return None + + watch_id = raw.get("id") + if _is_blank(watch_id): + problems.append(f"{label}: 'id' is required and must be a non-empty string") + watch_id = None + elif not _ID_RE.match(str(watch_id).strip()): + problems.append( + f"{label}: id '{watch_id}' is not a slug -- lowercase letters, digits, '-', '_', '.'" + ) + watch_id = None + else: + watch_id = str(watch_id).strip() + if watch_id in seen: + problems.append(f"{label}: duplicate watch id '{watch_id}'") + watch_id = None + else: + seen.add(watch_id) + + label = f"watches[{index}]" if watch_id is None else f"watch '{watch_id}'" + + unknown = sorted(set(raw) - WATCH_KEYS) + if unknown: + problems.append(f"{label}: unknown field(s) {unknown}; allowed: {sorted(WATCH_KEYS)}") + + claim = raw.get("claim") + if _is_blank(claim): + problems.append(f"{label}: 'claim' is required and must be a non-empty string") + + belief = _validate_belief(label, raw.get("belief"), problems) + so_what = _validate_so_what(label, raw.get("so_what"), position, problems) + triggers = _validate_triggers(label, raw.get("triggers"), problems) + staleness = _validate_staleness(label, raw.get("staleness_alert_days"), problems) + + expires = None + if "expires" not in raw or raw.get("expires") is None: + problems.append( + f"{label}: 'expires' is required -- a watch with no end date outlives the decision " + f"it serves and becomes a subscription." + ) + else: + expires = _as_date(raw.get("expires")) + if expires is None: + problems.append(f"{label}: expires {raw.get('expires')!r} is not a date (YYYY-MM-DD)") + elif expires < today: + problems.append( + f"{label}: expires {expires.isoformat()} is in the past " + f"({(today - expires).days}d ago). Extend it or delete the watch." + ) + expires = None + + if ( + watch_id is None + or _is_blank(claim) + or belief is None + or so_what is None + or not triggers + or expires is None + or staleness is None + ): + return None + + decision_key, because = so_what + return { + "id": watch_id, + "claim": str(claim).strip(), + "belief": belief, + "decision_key": decision_key, + "so_what": because, + "triggers": triggers, + "expires": expires, + "staleness_alert_days": staleness, + } diff --git a/src/position/watches.py b/src/position/watches.py index 9fd46fd..cb04cb7 100644 --- a/src/position/watches.py +++ b/src/position/watches.py @@ -51,7 +51,6 @@ from __future__ import annotations import logging -import re from dataclasses import dataclass, field from datetime import date from pathlib import Path @@ -59,9 +58,8 @@ import yaml -from src.utils.cadence import InvalidCadence, parse_cadence - -from .position import Position, _as_date, _is_blank +from ._validate import TRIGGER_FIELDS, TriggerClause, validate_watch +from .position import Position __all__ = [ "TRIGGER_FIELDS", @@ -74,24 +72,7 @@ logger = logging.getLogger(__name__) -_ID_RE = re.compile(r"^[a-z0-9][a-z0-9._-]*$") - _TOP_LEVEL_KEYS = {"version", "watches"} -_WATCH_KEYS = { - "id", - "claim", - "belief", - "so_what", - "triggers", - "expires", - "staleness_alert_days", -} -_SO_WHAT_KEYS = {"decision", "because"} - -# The only three axes a trigger clause may constrain. Each maps onto something -# Tier 1 can evaluate without a model: a word-boundary term match, an entity's -# declared surface forms, and the adapter that produced the observation. -TRIGGER_FIELDS = ("terms", "entities", "sources") class WatchError(ValueError): @@ -111,30 +92,6 @@ def __init__(self, path: Path, problems: list[str]) -> None: super().__init__(f"{path} is not a valid watch set:\n{body}") -@dataclass(frozen=True) -class TriggerClause: - """ - One conjunctive clause of a trigger. - - A clause fires when every field it populates matches: any of its ``terms`` - appears, **and** any of its ``entities`` appears, **and** the observation - came from one of its ``sources``. Fields left out are not constraints. The - watch fires when any clause fires, so ``triggers`` as a whole is a - disjunction of conjunctions -- expressive enough to say "FedRAMP *and* - continuous monitoring, or anything about it in the Federal Register", and - small enough that Tier 1 compiles it to regexes and a set membership test - with no interpreter in between. - """ - - terms: tuple[str, ...] = () - entities: tuple[str, ...] = () - sources: tuple[str, ...] = () - - def as_dict(self) -> dict[str, list[str]]: - """The JSON shape stored in ``watches.triggers``. Empty fields are omitted.""" - return {name: list(getattr(self, name)) for name in TRIGGER_FIELDS if getattr(self, name)} - - @dataclass(frozen=True) class Watch: """One pre-registered claim. Constructed only by :func:`load_watches`.""" @@ -163,303 +120,6 @@ def default_watches_path() -> Path: return Path(settings.watches_path).expanduser() -def _validate_so_what( - label: str, raw: Any, position: Position, problems: list[str] -) -> tuple[str, str] | None: - """ - Invariant 2, enforced. Returns ``(decision_key, because)`` or None. - - Every branch here rejects. None of them defaults, substitutes a placeholder, - or falls back to the claim text -- a Watch with no articulable decision is - curiosity, and the point of the invariant is that curiosity does not get - stored and then quietly alert. - """ - if raw is None: - problems.append( - f"{label}: 'so_what' is required and names the decision this watch serves. " - f"A watch with no decision attached is curiosity, not monitoring." - ) - return None - if isinstance(raw, str): - if not raw.strip(): - problems.append( - f"{label}: 'so_what' is empty. It must name a decision from the Position and " - f"say what changes; blank is rejected, never defaulted." - ) - else: - problems.append( - f"{label}: 'so_what' must be a mapping of 'decision' (a key from the Position) " - f"and 'because' (what changes), not a bare string." - ) - return None - if not isinstance(raw, dict): - problems.append(f"{label}: 'so_what' must be a mapping, got {type(raw).__name__}") - return None - - unknown = sorted(set(raw) - _SO_WHAT_KEYS) - if unknown: - problems.append( - f"{label}: so_what has unknown field(s) {unknown}; allowed: ['because', 'decision']" - ) - - decision_key = raw.get("decision") - because = raw.get("because") - ok = True - - if _is_blank(decision_key): - problems.append( - f"{label}: so_what.decision is required and must name a decision key from the " - f"Position. Known keys: {sorted(position.decision_keys)}" - ) - ok = False - elif str(decision_key).strip() not in position.decision_keys: - problems.append( - f"{label}: so_what.decision '{str(decision_key).strip()}' names no decision in " - f"{position.path}. Known keys: {sorted(position.decision_keys)}. Add the decision " - f"to the Position, or drop the watch." - ) - ok = False - - if _is_blank(because): - problems.append( - f"{label}: so_what.because is required and must be a non-empty sentence saying what " - f"changes for that decision if the claim resolves." - ) - ok = False - - if not ok: - return None - return str(decision_key).strip(), str(because).strip() - - -def _validate_string_list( - label: str, field_name: str, raw: Any, problems: list[str] -) -> tuple[str, ...]: - if raw is None: - return () - if isinstance(raw, str): - problems.append( - f"{label}: trigger '{field_name}' must be a list of terms, not a string. " - f"A trigger is compiled into a predicate, so it cannot be a sentence." - ) - return () - if not isinstance(raw, list): - problems.append(f"{label}: trigger '{field_name}' must be a list, got {type(raw).__name__}") - return () - values: list[str] = [] - for item in raw: - if _is_blank(item): - problems.append(f"{label}: trigger '{field_name}' contains an empty entry") - continue - values.append(str(item).strip()) - return tuple(values) - - -def _validate_triggers(label: str, raw: Any, problems: list[str]) -> tuple[TriggerClause, ...]: - """ - Structure, not prose. Every rejection here is a rejection of prose. - - ``trigger_condition`` in the deleted ledger accepted free text, and 33 rows - were written against it and graded zero times, because nothing downstream - could evaluate a sentence. Tier 1 compiles these clauses; a clause it cannot - compile is refused at the door rather than stored and skipped. - """ - if raw is None: - problems.append( - f"{label}: 'triggers' is required -- what observation would be candidate evidence?" - ) - return () - if isinstance(raw, str): - problems.append( - f"{label}: 'triggers' is prose. It must be a list of clauses over " - f"{list(TRIGGER_FIELDS)}, because Tier 1 compiles triggers into a deterministic " - f"predicate and cannot compile a sentence." - ) - return () - if not isinstance(raw, list): - problems.append(f"{label}: 'triggers' must be a list, got {type(raw).__name__}") - return () - if not raw: - problems.append(f"{label}: 'triggers' is empty -- a watch nothing can trigger never fires") - return () - - clauses: list[TriggerClause] = [] - for index, entry in enumerate(raw): - clause_label = f"{label}.triggers[{index}]" - if isinstance(entry, str): - problems.append( - f"{clause_label}: a trigger clause is a mapping over {list(TRIGGER_FIELDS)}, " - f"not the sentence {entry.strip()[:60]!r}. Prose triggers are what left 33 " - f"predictions ungradeable; write the terms and entities that would match." - ) - continue - if not isinstance(entry, dict): - problems.append(f"{clause_label}: expected a mapping, got {type(entry).__name__}") - continue - unknown = sorted(set(entry) - set(TRIGGER_FIELDS)) - if unknown: - problems.append( - f"{clause_label}: unknown field(s) {unknown}; a clause may constrain only " - f"{list(TRIGGER_FIELDS)}" - ) - continue - clause = TriggerClause( - terms=_validate_string_list(clause_label, "terms", entry.get("terms"), problems), - entities=_validate_string_list( - clause_label, "entities", entry.get("entities"), problems - ), - sources=_validate_string_list(clause_label, "sources", entry.get("sources"), problems), - ) - if not (clause.terms or clause.entities or clause.sources): - problems.append( - f"{clause_label}: constrains nothing. A clause with no terms, entities or " - f"sources matches every observation." - ) - continue - clauses.append(clause) - return tuple(clauses) - - -def _validate_belief(label: str, raw: Any, problems: list[str]) -> float | None: - if raw is None: - problems.append(f"{label}: 'belief' is required -- a probability in [0.0, 1.0]") - return None - if isinstance(raw, bool) or not isinstance(raw, int | float): - problems.append(f"{label}: belief must be a number in [0.0, 1.0], got {raw!r}") - return None - value = float(raw) - if not (0.0 <= value <= 1.0): - problems.append(f"{label}: belief {value} is outside [0.0, 1.0]") - return None - return value - - -def _validate_staleness(label: str, raw: Any, problems: list[str]) -> int | None: - """ - Days, written as an integer or as a cadence (``2w``, ``30d``). - - The cadence spelling goes through :mod:`src.utils.cadence` rather than a - second parser, so the whole system reads one interval grammar. - """ - if raw is None: - problems.append( - f"{label}: 'staleness_alert_days' is required -- after how much silence is the " - f"silence itself the finding?" - ) - return None - if isinstance(raw, bool): - problems.append( - f"{label}: staleness_alert_days must be a whole number of days, got {raw!r}" - ) - return None - if isinstance(raw, str): - try: - days = parse_cadence(raw).days - except InvalidCadence as exc: - problems.append(f"{label}: staleness_alert_days {raw!r} is not an interval -- {exc}") - return None - elif isinstance(raw, int): - days = raw - else: - problems.append( - f"{label}: staleness_alert_days must be a whole number of days, got {raw!r}" - ) - return None - if days < 1: - problems.append( - f"{label}: staleness_alert_days is {days}; it must be at least 1. Zero would make " - f"every watch stale the moment it is loaded." - ) - return None - return days - - -def _validate_watch( - index: int, - raw: Any, - position: Position, - seen: set[str], - today: date, - problems: list[str], -) -> Watch | None: - label = f"watches[{index}]" - if not isinstance(raw, dict): - problems.append(f"{label}: expected a mapping, got {type(raw).__name__}") - return None - - watch_id = raw.get("id") - if _is_blank(watch_id): - problems.append(f"{label}: 'id' is required and must be a non-empty string") - watch_id = None - elif not _ID_RE.match(str(watch_id).strip()): - problems.append( - f"{label}: id '{watch_id}' is not a slug -- lowercase letters, digits, '-', '_', '.'" - ) - watch_id = None - else: - watch_id = str(watch_id).strip() - if watch_id in seen: - problems.append(f"{label}: duplicate watch id '{watch_id}'") - watch_id = None - else: - seen.add(watch_id) - - label = f"watches[{index}]" if watch_id is None else f"watch '{watch_id}'" - - unknown = sorted(set(raw) - _WATCH_KEYS) - if unknown: - problems.append(f"{label}: unknown field(s) {unknown}; allowed: {sorted(_WATCH_KEYS)}") - - claim = raw.get("claim") - if _is_blank(claim): - problems.append(f"{label}: 'claim' is required and must be a non-empty string") - - belief = _validate_belief(label, raw.get("belief"), problems) - so_what = _validate_so_what(label, raw.get("so_what"), position, problems) - triggers = _validate_triggers(label, raw.get("triggers"), problems) - staleness = _validate_staleness(label, raw.get("staleness_alert_days"), problems) - - expires = None - if "expires" not in raw or raw.get("expires") is None: - problems.append( - f"{label}: 'expires' is required -- a watch with no end date outlives the decision " - f"it serves and becomes a subscription." - ) - else: - expires = _as_date(raw.get("expires")) - if expires is None: - problems.append(f"{label}: expires {raw.get('expires')!r} is not a date (YYYY-MM-DD)") - elif expires < today: - problems.append( - f"{label}: expires {expires.isoformat()} is in the past " - f"({(today - expires).days}d ago). Extend it or delete the watch." - ) - expires = None - - if ( - watch_id is None - or _is_blank(claim) - or belief is None - or so_what is None - or not triggers - or expires is None - or staleness is None - ): - return None - - decision_key, because = so_what - return Watch( - id=watch_id, - claim=str(claim).strip(), - belief=belief, - decision_key=decision_key, - so_what=because, - triggers=triggers, - expires=expires, - staleness_alert_days=staleness, - ) - - def load_watches( path: str | Path | None = None, *, @@ -506,27 +166,14 @@ def load_watches( seen: set[str] = set() stamp = today or date.today() for index, entry in enumerate(entries): - watch = _validate_watch(index, entry, position, seen, stamp, problems) - if watch is not None: - watches.append(watch) + fields = validate_watch(index, entry, position, seen, stamp, problems) + if fields is not None: + watches.append(Watch(**fields, source_path=target)) if problems: raise WatchError(target, problems) - return [ - Watch( - id=w.id, - claim=w.claim, - belief=w.belief, - decision_key=w.decision_key, - so_what=w.so_what, - triggers=w.triggers, - expires=w.expires, - staleness_alert_days=w.staleness_alert_days, - source_path=target, - ) - for w in watches - ] + return watches def sync_watches(session: Any, watches: list[Watch]) -> dict[str, list[str]]: From a6b9b4e68b33b17f57dbfe3a1606f29003582def Mon Sep 17 00:00:00 2001 From: sam-aydlette Date: Mon, 31 Aug 2026 12:00:37 -0400 Subject: [PATCH 3/3] docs: move the sync-delete caveat where 017 will see it The reviewer's one actionable finding: the caveat that sync_watches deletes rows for removed watches, and that task 017 will hang belief history off those rows, was disclosed only in the IC's report. That is the same failure as the cadence gap in task 011 -- a limitation that lives in a conversation is one nobody reads. It is now in sync_watches' own docstring and in backlog/017's landmines, which is where the implementer who would destroy calibration history will actually be looking. --- backlog/017-tier3-state-machine.md | 2 +- src/position/watches.py | 9 +++++++++ 2 files changed, 10 insertions(+), 1 deletion(-) diff --git a/backlog/017-tier3-state-machine.md b/backlog/017-tier3-state-machine.md index 7a7a004..2f73e72 100644 --- a/backlog/017-tier3-state-machine.md +++ b/backlog/017-tier3-state-machine.md @@ -4,7 +4,7 @@ STATUS: QUEUED # QUEUED | IN_PROGRESS | PARKED | DONE | FAILED SIZE: large ACCEPTANCE: `make check` passes, plus: belief updates and trigger evaluation are deterministic with **no LLM call**, proved the same way as Tier 1; hysteresis requires N independent sources or one source on a per-watch pre-declared authoritative list, and `N` is per-watch configuration, not a global constant; a debounce window collapses the same story arriving from many outlets into **one** transition, with a test driving twelve near-duplicate observations through and asserting exactly one Alert; a belief drift of 0.35 to 0.41 produces **no** Alert, asserted directly as the brief specifies; an Alert is created only on a state transition, never on evidence arrival; transitions are append-only and a Watch's belief history is reconstructable; and a test asserts that replaying the same Evidence twice produces the same transitions and no duplicate Alerts. OUT OF SCOPE: Sending anything -- Alerts are created here and delivered by task 019. Silencing or grouping UI. Resolution scoring; that is task 021. Auto-adjusting hysteresis from observed alert volume, which would let the system tune away its own noise complaint. Belief models more elaborate than what the acceptance requires -- no Bayesian update framework unless a specific watch demonstrably needs one, and if one does, park and say so. -LANDMINES: **This is the task where the rewrite's whole property lives or dies, and every failure mode here is silent.** Hysteresis too weak floods; too strong means the system never fires and looks identical to a broken pipeline -- which is why task 018 must land alongside, not after. **"N independent sources" needs a definition of independent that survives contact with reality**: three outlets syndicating one wire story are not three sources, and the near-duplicate grouping from task 014 is what makes the distinction available -- if it is not wired in here, "independent" degrades to "distinct feed URL" and the debounce is defeated by exactly the case it exists for. **The state machine must be a pure function of (prior state, evidence) or the append-only history is not reconstructable.** Reading current belief from a mutable column and writing back in place will pass tests and lose the audit trail. `main` is protected -- open a PR. +LANDMINES: **`sync_watches` currently DELETES the row for a watch removed from the file** (task 013, `src/position/watches.py`). That is correct while the table holds nothing derived. The moment this task hangs belief history off those rows, a delete destroys the record of what the operator believed and when -- which is the calibration data the system exists to accumulate. Change it to a soft retire before wiring history, or the first watch someone removes takes its own history with it and nothing surfaces the loss. **This is the task where the rewrite's whole property lives or dies, and every failure mode here is silent.** Hysteresis too weak floods; too strong means the system never fires and looks identical to a broken pipeline -- which is why task 018 must land alongside, not after. **"N independent sources" needs a definition of independent that survives contact with reality**: three outlets syndicating one wire story are not three sources, and the near-duplicate grouping from task 014 is what makes the distinction available -- if it is not wired in here, "independent" degrades to "distinct feed URL" and the debounce is defeated by exactly the case it exists for. **The state machine must be a pure function of (prior state, evidence) or the append-only history is not reconstructable.** Reading current belief from a mutable column and writing back in place will pass tests and lose the audit trail. `main` is protected -- open a PR. --- Written 2026-08-31. **The largest and most load-bearing task in this plan.** diff --git a/src/position/watches.py b/src/position/watches.py index cb04cb7..3d531a9 100644 --- a/src/position/watches.py +++ b/src/position/watches.py @@ -187,6 +187,15 @@ def sync_watches(session: Any, watches: list[Watch]) -> dict[str, list[str]]: Takes already-validated :class:`Watch` values, so an invalid file cannot reach this function at all. + + + Removing a watch from the file deletes its row. That is correct while this + table holds nothing derived -- the file is the source of truth and a watch + you deleted is a watch you stopped holding. **Task 017 hangs belief history + off these rows**, and at that point a delete destroys the record of what you + believed and when, which is the calibration data the system exists to + accumulate. Revisit this before 017 lands: the likely answer is a soft + retire rather than a delete. Noted 2026-08-31 (task 013). """ from src.database.models import Watch as WatchRow