docs(v0.4.0): freeze processing contracts for history, schedules, alerts, and candidates

SPEC.md

@@ -4,15 +4,17 @@
 
 This document is the living assignment truth source for SignalScope.
 
-It was reconciled again for the implemented `v0.3.0` ingestion baseline. The repository now ships a real fetch runtime and persists first-class observations in SQLite. Anything not implemented must still be listed here explicitly so the assignment remains honest.
+It was reconciled again after the implemented `v0.3.0` ingestion baseline and the documented `v0.4.0` processing-contract freeze. The repository currently ships a real fetch runtime with persisted `fetch_runs`, `metric_snapshots`, and `content_items`.
+
+The names and purposes of `schedule_runs`, `alert_events`, and `candidate_suggestions` are now committed for `v0.4.0`, but those tables and runtime paths are **not implemented yet**.
 
 ## 1. Product summary
 
 SignalScope is a config-driven competitive intelligence backend for tracking public signals, historical metrics, feed items, and operational fetch history across organizations, products, networks, initiatives, and open-source projects.
 
 The default demo dataset is based on Blockstream and related competitors/comparables, but the architecture must remain reusable through configuration alone.
 
-## 2. Current implementation truth (`v0.3.0` baseline)
+## 2. Current implementation truth (`v0.3.0` runtime baseline)
 
 ### Implemented now
 
@@ -49,7 +51,7 @@ The default demo dataset is based on Blockstream and related competitors/compara
 | Dashboard pages and visualizations | not implemented |
 | Owner-scoped `github.contributors` aggregation | intentionally deferred |
 
-## 3. Implemented ingestion baseline
+## 3. Runtime model and committed `v0.4.0` backend target
 
 ### 3.1 CLI execution model
 
@@ -63,15 +65,27 @@ The current operational modes are:
 | `serve` | run the API and dashboard server | reserved |
 | `schedule` | run configured jobs continuously | reserved |
 
-Important runtime details:
+Important current runtime details:
 
 - `fetch --source` selects by **source kind** (`github`, `news_rss`, `changelog`), not by config source ID.
 - `fetch --binding` can narrow execution to one configured binding.
-- the current CLI bootstraps and reseeds the selected config pack before each `fetch` invocation
-- because reseeding replaces dataset-scoped rows, reusing the same SQLite path across multiple CLI `fetch` invocations replaces earlier ingestion observations for that dataset
-- `fetch` exits with status code `1` when any selected binding fails, while still recording per-binding outcomes in `fetch_runs`
+- the current CLI bootstraps and reseeds the selected config pack before each `fetch` invocation.
+- because reseeding replaces dataset-scoped rows, repeated `fetch` invocations against the same SQLite path do **not** preserve prior ingestion history yet.
+- `fetch` exits with status code `1` when any selected binding fails, while still recording per-binding outcomes in `fetch_runs`.
+
+### 3.2 Committed `v0.4.0` backend target
+
+The committed backend-only `v0.4.0` milestone is expected to:
+
+- preserve repeated same-DB history in `fetch_runs`, `metric_snapshots`, and deduplicated `content_items`
+- execute due `schedule_jobs` through a deterministic one-shot scheduler path with persisted `schedule_runs`
+- evaluate persisted observations into materialized `alert_events`
+- persist repeated untracked co-mentions as reviewer-facing `candidate_suggestions`
+- keep API and dashboard work deferred beyond this milestone
 
-### 3.2 Persisted observation records
+## 4. Persisted observation model
+
+### 4.1 Implemented tables
 
 The first-class observation tables implemented in SQLite are:
 
@@ -95,6 +109,12 @@ Lifecycle rules in the current implementation:
 - `finished_at` is required for terminal rows
 - `records_written`, `metrics_written`, and `content_items_written` are stored on terminal rows
 
+History-preserving semantics committed for `v0.4.0`:
+
+- later manual fetches and future scheduler-triggered fetches append new rows instead of replacing earlier `fetch_runs`
+- the only allowed mutation of an existing row remains the in-place transition from `running` to a terminal status
+- successful and failed executions both remain part of the permanent operational history
+
 #### `metric_snapshots`
 
 `metric_snapshots` stores append-only metric observations with:
@@ -116,7 +136,14 @@ The current runtime uses the table as follows:
 - `news.article_count` leaves `scope_key` empty
 
 Implementation note:
-the schema supports `window_key`, but the current GitHub runtime encodes commit lookback directly in the metric key name (`github.commits_30d`, `github.commits_90d`) and therefore leaves `window_key` at its default `point_in_time`.
+
+- the schema supports `window_key`, but the current GitHub runtime encodes commit lookback directly in the metric key name (`github.commits_30d`, `github.commits_90d`) and therefore leaves `window_key` at its default `point_in_time`
+
+History-preserving semantics committed for `v0.4.0`:
+
+- snapshots remain append-only across repeated successful runs
+- a rerun that observes the same metric value still persists a new row when `captured_at` is new
+- later processing must derive deltas from ordered history rather than overwrite a synthetic latest-value row in place
 
 #### `content_items`
 
@@ -148,9 +175,53 @@ Deduplication in the current implementation is binding-level and first-write-win
 
 Cross-binding reconciliation remains deferred.
 
-### 3.3 Source-specific behavior
+History-preserving semantics committed for `v0.4.0`:
+
+- previously seen dedupe hits remain ignored without mutating the first stored row
+- newly discovered items append new rows
+- later alert and candidate processing must count distinct persisted evidence, not processor passes
+
+### 4.2 Committed-but-not-yet-implemented tables
+
+#### `schedule_runs`
 
-#### GitHub
+`schedule_runs` is the committed job-level execution history for one due `schedule_jobs` execution.
+
+Frozen `v0.4.0` contract:
+
+- one row represents one due execution of one `schedule_jobs` row
+- not-due checks do **not** persist synthetic rows
+- the canonical job-level statuses are `running`, `succeeded`, `partial`, and `failed`
+- `schedule_runs` summarize scheduler orchestration, while `fetch_runs` retain per-binding detail
+- repeated scheduler executions append new rows; they do not replace earlier scheduler history
+
+#### `alert_events`
+
+`alert_events` is the committed materialized alert history generated from persisted observations and configured `alert_rules`.
+
+Frozen `v0.4.0` contract:
+
+- metric-based rules evaluate persisted `metric_snapshots`
+- `event_key = "release.published"` evaluates persisted `content_items` with `item_type = "release"`
+- `alert_rule_id`, `source_id`, and `severity` are copied from the referenced configured rule into the derived row
+- reevaluating unchanged evidence must not create duplicate open rows for the same logical alert instance
+- later distinct recurrences may create new rows after cooldown/resolution rules are satisfied
+
+#### `candidate_suggestions`
+
+`candidate_suggestions` is the committed reviewer-facing queue for repeated untracked co-mentions discovered from persisted evidence.
+
+Frozen `v0.4.0` contract:
+
+- the canonical source event for the initial baseline is `event_key = "news.co_mention"`
+- the initial threshold rule shape remains `condition.type = "candidate_repeat_gte"`
+- a suggestion row is created when repeated distinct persisted evidence crosses threshold
+- rerunning processing without new evidence must not create duplicate suggestion rows or inflate `mention_count`
+- accepting a suggestion does **not** imply automatic tracked-entity creation in `v0.4.0`
+
+## 5. Current source-specific behavior
+
+### GitHub
 
 The current GitHub runtime supports two scope forms.
 
@@ -181,9 +252,9 @@ Important GitHub runtime notes:
 - when `GITHUB_TOKEN` is present, GitHub-backed requests use it automatically
 - owner-scoped aggregates currently include public archived repositories and public forks
 - private repositories remain out of scope
-- the current runtime also treats missing latest-release data as a binding failure for repo-scoped GitHub metrics, and owner-scoped aggregation fails if no included public repository exposes any latest release
+- the current runtime treats missing latest-release data as a binding failure for repo-scoped GitHub metrics, and owner-scoped aggregation fails if no included public repository exposes any latest release
 
-#### News RSS
+### News RSS
 
 The current query-based news provider is **Google News RSS search**.
 
@@ -193,7 +264,7 @@ Successful news runs persist:
 - one `news.article_count` metric snapshot
 - zero or more `news_article` content items after in-run dedupe and optional `max_items` limiting
 
-#### Changelog
+### Changelog
 
 The changelog fetcher supports:
 
@@ -206,7 +277,7 @@ The changelog fetcher supports:
 
 GitHub-backed changelog requests also honor optional `GITHUB_TOKEN` auth when present.
 
-## 4. Current schema subset
+## 6. Current schema subset
 
 ### Implemented tables
 
@@ -222,36 +293,37 @@ GitHub-backed changelog requests also honor optional `GITHUB_TOKEN` auth when pr
 - `metric_snapshots`
 - `content_items`
 
-### Deferred tables
+### Committed `v0.4.0` tables not implemented yet
 
-The following tables remain conceptual or deferred:
+The following persisted tables are committed for `v0.4.0`, but are not implemented yet:
 
 - optional `source_records`
+- `schedule_runs`
 - `alert_events`
-- `candidate_entities`
+- `candidate_suggestions`
 
-## 5. Assignment-layer status
+## 7. Assignment-layer status
 
 | Layer | Brief expectation | Current status |
 | --- | --- | --- |
 | Layer 1: data ingestion | at least three working fetchers, including GitHub | baseline delivered |
 | Layer 2: storage and processing | persistence, scheduled refresh, delta detection, alerts, candidate discovery | partially delivered through SQLite ingestion persistence only |
 | Layer 3: frontend/dashboard | overview, drill-down, compare, feed, alerts | not implemented |
 
-## 6. Deferred feature register
+## 8. Deferred feature register
 
 | Item | Status | Implementation note |
 | --- | --- | --- |
 | App-store ingestion | deferred beyond `v0.3.0` | source kinds remain reserved in config but no runtime exists yet |
 | Social ingestion | deferred beyond `v0.3.0` | source kind is reserved; no runtime exists yet |
 | Owner-scoped `github.contributors` | deferred | must not be implied by owner-level GitHub support |
-| Scheduler runtime | deferred to `v0.4.0` | should execute enabled jobs by cadence over persisted state |
-| Delta detection and alert materialization | deferred to `v0.4.0` | must evaluate rules against persisted observations rather than live upstream responses |
-| Candidate competitor discovery | deferred to `v0.4.0` | should surface repeated co-mentions for manual review only |
+| Scheduler runtime | deferred to `v0.4.0` | should persist `schedule_runs` and execute due jobs without replacing prior observation history |
+| Delta detection and alert materialization | deferred to `v0.4.0` | should persist `alert_events` linked to configured rules and evidence timestamps |
+| Candidate competitor discovery | deferred to `v0.4.0` | should persist reviewer-facing `candidate_suggestions` from repeated news co-mentions |
 | Read-only API | deferred to `v0.5.0` | must read persisted SQLite data, not live sources |
 | Dashboard delivery | deferred to `v0.5.0` | should reuse the same read models as the API |
 
-## 7. Non-functional requirements
+## 9. Non-functional requirements
 
 | Requirement | Expectation |
 | --- | --- |
@@ -262,7 +334,7 @@ The following tables remain conceptual or deferred:
 | transparency | fetch failures and counts must be inspectable |
 | reusability | no application logic should depend on Blockstream-specific names |
 
-## 8. Delivery policy
+## 10. Delivery policy
 
 SignalScope follows a simple policy:
 
@@ -271,7 +343,7 @@ SignalScope follows a simple policy:
 3. verify the contract locally and in CI
 4. update this `SPEC.md` whenever runtime truth changes
 
-## 9. References
+## 11. References
 
 - `README.md`
 - `docs/architecture.md`
@@ -286,3 +358,5 @@ SignalScope follows a simple policy:
 SignalScope now ships a real `v0.3.0` ingestion baseline: config loading, validation, SQLite bootstrap, `fetch` execution, and persisted `fetch_runs`, `metric_snapshots`, and `content_items`.
 
 Scheduler work, alerting, candidate discovery, API delivery, and dashboard UI remain intentionally deferred and must continue to be documented as such.
+
+The committed `v0.4.0` backend milestone is expected to add history-preserving repeated runs, `schedule_runs`, `alert_events`, and `candidate_suggestions` while keeping API and dashboard work deferred.

docs/configuration.md

@@ -11,6 +11,8 @@ The repository now implements:
 - `fetch` runtime support for `github`, `news_rss`, and `changelog`
 - persisted ingestion observations in SQLite
 
+This document also freezes how the existing `alerts.json` and `schedules.json` fields are intended to feed the planned `v0.4.0` processing layer. It does **not** imply that scheduler execution, alert evaluation, or candidate discovery are already implemented.
+
 This document therefore distinguishes between:
 
 - config rules enforced by the current codebase
@@ -233,15 +235,21 @@ This file defines source types and per-entity bindings.
   - `changelog`
 
 Operational note:
-the CLI `fetch --source` flag selects by canonical source kind (`github`, `news_rss`, `changelog`), while persisted `source_id` columns keep the configured source IDs from `sources.json`.
+
+- the CLI `fetch --source` flag selects by canonical source kind (`github`, `news_rss`, `changelog`), while persisted `source_id` columns keep the configured source IDs from `sources.json`
 
 ### GitHub binding scope
+
 A GitHub binding must define either:
+
 - a non-empty `repos` list
+
 or
+
 - an `owner` plus `owner_type`
 
 Supported `owner_type` values:
+
 - `organization`
 - `user`
 
@@ -294,30 +302,38 @@ Notes:
 - the current runtime derives `github.latest_release_published_at` from the latest available public release across the included repositories and fails the binding when no included public repository exposes any latest release
 
 ### News RSS binding scope
+
 A news RSS binding must define:
+
 - `query` as a non-empty string
 
 Optional:
+
 - `max_items` as a positive whole number
 
 #### Query provider choice for `v0.3.0`
+
 The binding contract remains query-driven and provider-agnostic at the config level:
+
 - `scope.query` is the canonical config input
 - `scope.max_items`, when present, limits the normalized items accepted from one provider response
 
 For the first live `v0.3.0` runtime, the documented query-based provider is the Google News RSS search endpoint.
 
 This means:
+
 - the runtime constructs requests from the configured query
 - the runtime issues Google News RSS search requests under `/rss/search`
 - accepted response entries normalize into `content_items` with `item_type = "news_article"`
-- each successful binding execution also persists one `news.article_count` snapshot using the post-dedupe accepted item count (after `max_items`, when configured)
+- each successful binding execution also persists one `news.article_count` snapshot using the post-dedupe accepted item count, after `max_items` when configured
 - alternative query-based providers such as NewsAPI are intentionally deferred
 - the current docs do **not** imply runtime provider selection from config
 
 Example:
+
 ```json
-{  "query": "Blockstream Jade",
+{
+  "query": "Blockstream Jade",
   "max_items": 15
 }
 ```
@@ -356,6 +372,7 @@ Examples:
 ```
 
 Normalization notes for `v0.3.0`:
+
 - `mode = "github_releases"` persists `content_items` as `release`
 - `mode = "rss"` persists `content_items` as `changelog_entry`
 
@@ -421,24 +438,46 @@ This file defines threshold and event rules used later by processors.
 ### Condition-type expectations currently enforced
 
 #### `absolute_change_gte`
+
 - metric-based
 - requires `condition.value`
 
 #### `percent_change_gte`
+
 - metric-based
 - requires `condition.value`
 
 #### `candidate_repeat_gte`
+
 - event-based
 - requires `condition.value`
 
 #### `silence_then_event`
+
 - event-based
 - requires `condition.silence_days`
 - if `window` is present, it must be expressed in `day` units and must match `condition.silence_days`
 
 That last rule removes ambiguity before the processing milestone begins.
 
+### Planned `v0.4.0` processing mapping (contract freeze, not yet implemented)
+
+The current validator already enforces the rule-shape fields documented in this section. The committed `v0.4.0` backend milestone is expected to consume them as follows:
+
+- `metric_key` rules evaluate persisted `metric_snapshots`
+- `event_key = "release.published"` evaluates persisted `content_items` with `item_type = "release"`
+- `event_key = "news.co_mention"` evaluates repeated co-mention evidence derived from persisted `content_items` with `item_type = "news_article"`
+- `window` is the lookback interval used for comparison or repetition detection
+- `condition.value` or `condition.silence_days` is the trigger threshold
+- `cooldown` suppresses duplicate `alert_events`, but must not erase prior alert history or block canonical `candidate_suggestions` updates
+
+No additional `alerts.json` fields are required for the initial `v0.4.0` baseline. Candidate suggestion heuristics reuse the existing `candidate_repeat_gte` rule shape rather than introducing a separate config file.
+
+Important honesty note:
+
+- the current validator does **not** yet whitelist concrete `event_key` strings
+- the values above are the committed `v0.4.0` processing keys for the current baseline docs and demo pack
+
 ## `schedules.json`
 
 This file defines refresh jobs per source.
@@ -475,6 +514,17 @@ This file defines refresh jobs per source.
 - `jitter_seconds` must be greater than or equal to zero
 - enabled jobs must reference enabled sources
 
+### Planned `v0.4.0` scheduler mapping (contract freeze, not yet implemented)
+
+No new schedule-job fields are required for the initial `v0.4.0` backend milestone.
+
+The existing job knobs map as follows:
+
+- `cadence` defines when a job becomes due
+- `timeout_seconds` is the intended execution deadline for one due job run
+- `jitter_seconds` remains part of the config contract, but the reviewer-friendly one-shot scheduler mode must remain deterministic and must not imply long-running daemon behavior automatically
+- each due execution should materialize one `schedule_runs` row and then fan out into the relevant `fetch_runs` for that source
+
 ## Config loading order
 
 The runtime loads config in this order: