diff --git a/SPEC.md b/SPEC.md index 878d956..3574158 100644 --- a/SPEC.md +++ b/SPEC.md @@ -4,7 +4,9 @@ This document is the living assignment truth source for SignalScope. -It was reconciled again for the implemented `v0.3.0` ingestion baseline. The repository now ships a real fetch runtime and persists first-class observations in SQLite. Anything not implemented must still be listed here explicitly so the assignment remains honest. +It was reconciled again after the implemented `v0.3.0` ingestion baseline and the documented `v0.4.0` processing-contract freeze. The repository currently ships a real fetch runtime with persisted `fetch_runs`, `metric_snapshots`, and `content_items`. + +The names and purposes of `schedule_runs`, `alert_events`, and `candidate_suggestions` are now committed for `v0.4.0`, but those tables and runtime paths are **not implemented yet**. ## 1. Product summary @@ -12,7 +14,7 @@ SignalScope is a config-driven competitive intelligence backend for tracking pub The default demo dataset is based on Blockstream and related competitors/comparables, but the architecture must remain reusable through configuration alone. -## 2. Current implementation truth (`v0.3.0` baseline) +## 2. Current implementation truth (`v0.3.0` runtime baseline) ### Implemented now @@ -49,7 +51,7 @@ The default demo dataset is based on Blockstream and related competitors/compara | Dashboard pages and visualizations | not implemented | | Owner-scoped `github.contributors` aggregation | intentionally deferred | -## 3. Implemented ingestion baseline +## 3. Runtime model and committed `v0.4.0` backend target ### 3.1 CLI execution model @@ -63,15 +65,27 @@ The current operational modes are: | `serve` | run the API and dashboard server | reserved | | `schedule` | run configured jobs continuously | reserved | -Important runtime details: +Important current runtime details: - `fetch --source` selects by **source kind** (`github`, `news_rss`, `changelog`), not by config source ID. - `fetch --binding` can narrow execution to one configured binding. -- the current CLI bootstraps and reseeds the selected config pack before each `fetch` invocation -- because reseeding replaces dataset-scoped rows, reusing the same SQLite path across multiple CLI `fetch` invocations replaces earlier ingestion observations for that dataset -- `fetch` exits with status code `1` when any selected binding fails, while still recording per-binding outcomes in `fetch_runs` +- the current CLI bootstraps and reseeds the selected config pack before each `fetch` invocation. +- because reseeding replaces dataset-scoped rows, repeated `fetch` invocations against the same SQLite path do **not** preserve prior ingestion history yet. +- `fetch` exits with status code `1` when any selected binding fails, while still recording per-binding outcomes in `fetch_runs`. + +### 3.2 Committed `v0.4.0` backend target + +The committed backend-only `v0.4.0` milestone is expected to: + +- preserve repeated same-DB history in `fetch_runs`, `metric_snapshots`, and deduplicated `content_items` +- execute due `schedule_jobs` through a deterministic one-shot scheduler path with persisted `schedule_runs` +- evaluate persisted observations into materialized `alert_events` +- persist repeated untracked co-mentions as reviewer-facing `candidate_suggestions` +- keep API and dashboard work deferred beyond this milestone -### 3.2 Persisted observation records +## 4. Persisted observation model + +### 4.1 Implemented tables The first-class observation tables implemented in SQLite are: @@ -95,6 +109,12 @@ Lifecycle rules in the current implementation: - `finished_at` is required for terminal rows - `records_written`, `metrics_written`, and `content_items_written` are stored on terminal rows +History-preserving semantics committed for `v0.4.0`: + +- later manual fetches and future scheduler-triggered fetches append new rows instead of replacing earlier `fetch_runs` +- the only allowed mutation of an existing row remains the in-place transition from `running` to a terminal status +- successful and failed executions both remain part of the permanent operational history + #### `metric_snapshots` `metric_snapshots` stores append-only metric observations with: @@ -116,7 +136,14 @@ The current runtime uses the table as follows: - `news.article_count` leaves `scope_key` empty Implementation note: -the schema supports `window_key`, but the current GitHub runtime encodes commit lookback directly in the metric key name (`github.commits_30d`, `github.commits_90d`) and therefore leaves `window_key` at its default `point_in_time`. + +- the schema supports `window_key`, but the current GitHub runtime encodes commit lookback directly in the metric key name (`github.commits_30d`, `github.commits_90d`) and therefore leaves `window_key` at its default `point_in_time` + +History-preserving semantics committed for `v0.4.0`: + +- snapshots remain append-only across repeated successful runs +- a rerun that observes the same metric value still persists a new row when `captured_at` is new +- later processing must derive deltas from ordered history rather than overwrite a synthetic latest-value row in place #### `content_items` @@ -148,9 +175,53 @@ Deduplication in the current implementation is binding-level and first-write-win Cross-binding reconciliation remains deferred. -### 3.3 Source-specific behavior +History-preserving semantics committed for `v0.4.0`: + +- previously seen dedupe hits remain ignored without mutating the first stored row +- newly discovered items append new rows +- later alert and candidate processing must count distinct persisted evidence, not processor passes + +### 4.2 Committed-but-not-yet-implemented tables + +#### `schedule_runs` -#### GitHub +`schedule_runs` is the committed job-level execution history for one due `schedule_jobs` execution. + +Frozen `v0.4.0` contract: + +- one row represents one due execution of one `schedule_jobs` row +- not-due checks do **not** persist synthetic rows +- the canonical job-level statuses are `running`, `succeeded`, `partial`, and `failed` +- `schedule_runs` summarize scheduler orchestration, while `fetch_runs` retain per-binding detail +- repeated scheduler executions append new rows; they do not replace earlier scheduler history + +#### `alert_events` + +`alert_events` is the committed materialized alert history generated from persisted observations and configured `alert_rules`. + +Frozen `v0.4.0` contract: + +- metric-based rules evaluate persisted `metric_snapshots` +- `event_key = "release.published"` evaluates persisted `content_items` with `item_type = "release"` +- `alert_rule_id`, `source_id`, and `severity` are copied from the referenced configured rule into the derived row +- reevaluating unchanged evidence must not create duplicate open rows for the same logical alert instance +- later distinct recurrences may create new rows after cooldown/resolution rules are satisfied + +#### `candidate_suggestions` + +`candidate_suggestions` is the committed reviewer-facing queue for repeated untracked co-mentions discovered from persisted evidence. + +Frozen `v0.4.0` contract: + +- the canonical source event for the initial baseline is `event_key = "news.co_mention"` +- the initial threshold rule shape remains `condition.type = "candidate_repeat_gte"` +- a suggestion row is created when repeated distinct persisted evidence crosses threshold +- rerunning processing without new evidence must not create duplicate suggestion rows or inflate `mention_count` +- accepting a suggestion does **not** imply automatic tracked-entity creation in `v0.4.0` + +## 5. Current source-specific behavior + +### GitHub The current GitHub runtime supports two scope forms. @@ -181,9 +252,9 @@ Important GitHub runtime notes: - when `GITHUB_TOKEN` is present, GitHub-backed requests use it automatically - owner-scoped aggregates currently include public archived repositories and public forks - private repositories remain out of scope -- the current runtime also treats missing latest-release data as a binding failure for repo-scoped GitHub metrics, and owner-scoped aggregation fails if no included public repository exposes any latest release +- the current runtime treats missing latest-release data as a binding failure for repo-scoped GitHub metrics, and owner-scoped aggregation fails if no included public repository exposes any latest release -#### News RSS +### News RSS The current query-based news provider is **Google News RSS search**. @@ -193,7 +264,7 @@ Successful news runs persist: - one `news.article_count` metric snapshot - zero or more `news_article` content items after in-run dedupe and optional `max_items` limiting -#### Changelog +### Changelog The changelog fetcher supports: @@ -206,7 +277,7 @@ The changelog fetcher supports: GitHub-backed changelog requests also honor optional `GITHUB_TOKEN` auth when present. -## 4. Current schema subset +## 6. Current schema subset ### Implemented tables @@ -222,15 +293,16 @@ GitHub-backed changelog requests also honor optional `GITHUB_TOKEN` auth when pr - `metric_snapshots` - `content_items` -### Deferred tables +### Committed `v0.4.0` tables not implemented yet -The following tables remain conceptual or deferred: +The following persisted tables are committed for `v0.4.0`, but are not implemented yet: - optional `source_records` +- `schedule_runs` - `alert_events` -- `candidate_entities` +- `candidate_suggestions` -## 5. Assignment-layer status +## 7. Assignment-layer status | Layer | Brief expectation | Current status | | --- | --- | --- | @@ -238,20 +310,20 @@ The following tables remain conceptual or deferred: | Layer 2: storage and processing | persistence, scheduled refresh, delta detection, alerts, candidate discovery | partially delivered through SQLite ingestion persistence only | | Layer 3: frontend/dashboard | overview, drill-down, compare, feed, alerts | not implemented | -## 6. Deferred feature register +## 8. Deferred feature register | Item | Status | Implementation note | | --- | --- | --- | | App-store ingestion | deferred beyond `v0.3.0` | source kinds remain reserved in config but no runtime exists yet | | Social ingestion | deferred beyond `v0.3.0` | source kind is reserved; no runtime exists yet | | Owner-scoped `github.contributors` | deferred | must not be implied by owner-level GitHub support | -| Scheduler runtime | deferred to `v0.4.0` | should execute enabled jobs by cadence over persisted state | -| Delta detection and alert materialization | deferred to `v0.4.0` | must evaluate rules against persisted observations rather than live upstream responses | -| Candidate competitor discovery | deferred to `v0.4.0` | should surface repeated co-mentions for manual review only | +| Scheduler runtime | deferred to `v0.4.0` | should persist `schedule_runs` and execute due jobs without replacing prior observation history | +| Delta detection and alert materialization | deferred to `v0.4.0` | should persist `alert_events` linked to configured rules and evidence timestamps | +| Candidate competitor discovery | deferred to `v0.4.0` | should persist reviewer-facing `candidate_suggestions` from repeated news co-mentions | | Read-only API | deferred to `v0.5.0` | must read persisted SQLite data, not live sources | | Dashboard delivery | deferred to `v0.5.0` | should reuse the same read models as the API | -## 7. Non-functional requirements +## 9. Non-functional requirements | Requirement | Expectation | | --- | --- | @@ -262,7 +334,7 @@ The following tables remain conceptual or deferred: | transparency | fetch failures and counts must be inspectable | | reusability | no application logic should depend on Blockstream-specific names | -## 8. Delivery policy +## 10. Delivery policy SignalScope follows a simple policy: @@ -271,7 +343,7 @@ SignalScope follows a simple policy: 3. verify the contract locally and in CI 4. update this `SPEC.md` whenever runtime truth changes -## 9. References +## 11. References - `README.md` - `docs/architecture.md` @@ -286,3 +358,5 @@ SignalScope follows a simple policy: SignalScope now ships a real `v0.3.0` ingestion baseline: config loading, validation, SQLite bootstrap, `fetch` execution, and persisted `fetch_runs`, `metric_snapshots`, and `content_items`. Scheduler work, alerting, candidate discovery, API delivery, and dashboard UI remain intentionally deferred and must continue to be documented as such. + +The committed `v0.4.0` backend milestone is expected to add history-preserving repeated runs, `schedule_runs`, `alert_events`, and `candidate_suggestions` while keeping API and dashboard work deferred. diff --git a/docs/configuration.md b/docs/configuration.md index d3a0ae5..9b4153e 100644 --- a/docs/configuration.md +++ b/docs/configuration.md @@ -11,6 +11,8 @@ The repository now implements: - `fetch` runtime support for `github`, `news_rss`, and `changelog` - persisted ingestion observations in SQLite +This document also freezes how the existing `alerts.json` and `schedules.json` fields are intended to feed the planned `v0.4.0` processing layer. It does **not** imply that scheduler execution, alert evaluation, or candidate discovery are already implemented. + This document therefore distinguishes between: - config rules enforced by the current codebase @@ -233,15 +235,21 @@ This file defines source types and per-entity bindings. - `changelog` Operational note: -the CLI `fetch --source` flag selects by canonical source kind (`github`, `news_rss`, `changelog`), while persisted `source_id` columns keep the configured source IDs from `sources.json`. + +- the CLI `fetch --source` flag selects by canonical source kind (`github`, `news_rss`, `changelog`), while persisted `source_id` columns keep the configured source IDs from `sources.json` ### GitHub binding scope + A GitHub binding must define either: + - a non-empty `repos` list + or + - an `owner` plus `owner_type` Supported `owner_type` values: + - `organization` - `user` @@ -294,30 +302,38 @@ Notes: - the current runtime derives `github.latest_release_published_at` from the latest available public release across the included repositories and fails the binding when no included public repository exposes any latest release ### News RSS binding scope + A news RSS binding must define: + - `query` as a non-empty string Optional: + - `max_items` as a positive whole number #### Query provider choice for `v0.3.0` + The binding contract remains query-driven and provider-agnostic at the config level: + - `scope.query` is the canonical config input - `scope.max_items`, when present, limits the normalized items accepted from one provider response For the first live `v0.3.0` runtime, the documented query-based provider is the Google News RSS search endpoint. This means: + - the runtime constructs requests from the configured query - the runtime issues Google News RSS search requests under `/rss/search` - accepted response entries normalize into `content_items` with `item_type = "news_article"` -- each successful binding execution also persists one `news.article_count` snapshot using the post-dedupe accepted item count (after `max_items`, when configured) +- each successful binding execution also persists one `news.article_count` snapshot using the post-dedupe accepted item count, after `max_items` when configured - alternative query-based providers such as NewsAPI are intentionally deferred - the current docs do **not** imply runtime provider selection from config Example: + ```json -{ "query": "Blockstream Jade", +{ + "query": "Blockstream Jade", "max_items": 15 } ``` @@ -356,6 +372,7 @@ Examples: ``` Normalization notes for `v0.3.0`: + - `mode = "github_releases"` persists `content_items` as `release` - `mode = "rss"` persists `content_items` as `changelog_entry` @@ -421,24 +438,46 @@ This file defines threshold and event rules used later by processors. ### Condition-type expectations currently enforced #### `absolute_change_gte` + - metric-based - requires `condition.value` #### `percent_change_gte` + - metric-based - requires `condition.value` #### `candidate_repeat_gte` + - event-based - requires `condition.value` #### `silence_then_event` + - event-based - requires `condition.silence_days` - if `window` is present, it must be expressed in `day` units and must match `condition.silence_days` That last rule removes ambiguity before the processing milestone begins. +### Planned `v0.4.0` processing mapping (contract freeze, not yet implemented) + +The current validator already enforces the rule-shape fields documented in this section. The committed `v0.4.0` backend milestone is expected to consume them as follows: + +- `metric_key` rules evaluate persisted `metric_snapshots` +- `event_key = "release.published"` evaluates persisted `content_items` with `item_type = "release"` +- `event_key = "news.co_mention"` evaluates repeated co-mention evidence derived from persisted `content_items` with `item_type = "news_article"` +- `window` is the lookback interval used for comparison or repetition detection +- `condition.value` or `condition.silence_days` is the trigger threshold +- `cooldown` suppresses duplicate `alert_events`, but must not erase prior alert history or block canonical `candidate_suggestions` updates + +No additional `alerts.json` fields are required for the initial `v0.4.0` baseline. Candidate suggestion heuristics reuse the existing `candidate_repeat_gte` rule shape rather than introducing a separate config file. + +Important honesty note: + +- the current validator does **not** yet whitelist concrete `event_key` strings +- the values above are the committed `v0.4.0` processing keys for the current baseline docs and demo pack + ## `schedules.json` This file defines refresh jobs per source. @@ -475,6 +514,17 @@ This file defines refresh jobs per source. - `jitter_seconds` must be greater than or equal to zero - enabled jobs must reference enabled sources +### Planned `v0.4.0` scheduler mapping (contract freeze, not yet implemented) + +No new schedule-job fields are required for the initial `v0.4.0` backend milestone. + +The existing job knobs map as follows: + +- `cadence` defines when a job becomes due +- `timeout_seconds` is the intended execution deadline for one due job run +- `jitter_seconds` remains part of the config contract, but the reviewer-friendly one-shot scheduler mode must remain deterministic and must not imply long-running daemon behavior automatically +- each due execution should materialize one `schedule_runs` row and then fan out into the relevant `fetch_runs` for that source + ## Config loading order The runtime loads config in this order: diff --git a/docs/data-model.md b/docs/data-model.md index b506a01..a95c3f5 100644 --- a/docs/data-model.md +++ b/docs/data-model.md @@ -2,18 +2,34 @@ ## Document status -This document was updated for the implemented `v0.3.0` ingestion baseline. +This document now serves two roles: -The repository now persists both the catalog/bootstrap dataset and first-class ingestion observations in SQLite. In particular, the following tables are real runtime artifacts rather than future-only contracts: +- describe the implemented `v0.3.0` ingestion write-side already persisted in SQLite +- freeze the committed `v0.4.0` backend-processing contracts before scheduler and derived-signal code lands + +The repository currently implements these write-side tables: - `fetch_runs` - `metric_snapshots` - `content_items` -Scheduler work, alert materialization, candidate discovery, API delivery, and dashboard UI remain intentionally deferred. +The following names and purposes are now frozen for `v0.4.0`, but are **not implemented yet**: + +- `schedule_runs` +- `alert_events` +- `candidate_suggestions` + +Important current-runtime boundary: + +- the current CLI still reseeds the selected dataset before every `fetch` +- because of that, repeated CLI `fetch` invocations against the same SQLite path do **not** preserve history yet +- the repeated-run semantics below therefore define the committed `v0.4.0` target, not already-delivered CLI behavior +- API delivery and dashboard UI remain intentionally deferred ## Modeling goals + The data model must: + - support organizations, products, networks, initiatives, and open-source projects - preserve historical observations so trends and deltas can be reproduced - separate source-specific fetch logic from normalized stored records @@ -21,7 +37,20 @@ The data model must: - support multiple source bindings per entity - remain generic so another organization can replace the Blockstream demo dataset through configuration only +## Repeated-run history semantics frozen for `v0.4.0` + +Reusing the same SQLite path for repeated fetches or one-shot scheduler executions must preserve earlier observation history rather than replacing it. + +The committed write-side semantics are: + +- `fetch_runs` append one row per binding execution; prior terminal rows remain intact +- `metric_snapshots` append one row per emitted observation even when the metric value repeats across runs +- `content_items` preserve first-write rows and only add previously unseen items according to the documented dedupe contract +- reprocessing the same persisted evidence must be idempotent: it must not duplicate alert events or candidate suggestions +- `seed-demo` remains a catalog/bootstrap operation and is not part of the normal history-preserving review flow + ## Entity taxonomy + SignalScope tracks a generic `entity` concept with a constrained `kind`. | Kind | Meaning | Examples | @@ -36,7 +65,9 @@ SignalScope tracks a generic `entity` concept with a constrained `kind`. This taxonomy is intentionally broader than “company” so the same dashboard can compare products and open-source ecosystems that do not map cleanly to one legal entity. ## Identity rules + Each entity should have: + - a stable internal `id` used in storage and configuration - a human-readable `slug` used in URLs and filters - a display `name` @@ -45,15 +76,19 @@ Each entity should have: - an optional `vertical` for reporting and grouping ### Identifier conventions + Recommended conventions: + - `id`: machine-oriented and stable, for example `product-jade` or `project-bdk` - `slug`: short URL-safe label, for example `jade` or `bitcoin-dev-kit` - `kind`: one of the supported taxonomy values above ## Relationship model + Relationships are directional in storage even when the UI treats them as symmetric. ### Initial relationship types + | Type | Meaning | | --- | --- | | `owns` | Parent organization owns or maintains the child entity | @@ -63,14 +98,17 @@ Relationships are directional in storage even when the UI treats them as symmetr For the default dataset, `owns` and `competes_with` remain the most important relationship types. ## Write-side storage concepts + The write side should be append-friendly and source-aware. All write-side records remain dataset-scoped even when the field lists below focus on ingestion-specific columns. ### 1. `entities` + Canonical tracked entities. Suggested fields: + - `id` - `slug` - `kind` @@ -84,9 +122,11 @@ Suggested fields: - `updated_at` ### 2. `entity_aliases` + Alternative names used for matching, searching, and news query generation. Suggested fields: + - `id` - `entity_id` - `alias` @@ -95,12 +135,15 @@ Suggested fields: - `is_primary` Uniqueness guidance: + - unique on `(entity_id, normalized_alias)` ### 3. `entity_relationships` + Directed relationships between tracked entities. Suggested fields: + - `id` - `source_entity_id` - `relation_type` @@ -109,12 +152,15 @@ Suggested fields: - `created_at` Uniqueness guidance: + - unique on `(source_entity_id, relation_type, target_entity_id)` ### 4. `source_bindings` + Normalized mapping between an entity and a public source. Suggested fields: + - `id` - `entity_id` - `source_id` @@ -125,6 +171,7 @@ Suggested fields: - `updated_at` Examples of source-specific scope: + - GitHub owner or repo allowlist - news query string - changelog RSS feed URL @@ -164,10 +211,19 @@ Current runtime notes: - terminal rows store `records_written`, `metrics_written`, and `content_items_written` - the CLI returns a non-zero exit status when any selected binding fails, but successful and failed binding rows are still persisted +History-preserving semantics frozen for `v0.4.0`: + +- one `fetch_runs` row still represents one binding execution +- later runs against the same SQLite path append new terminal rows; they do not replace earlier `fetch_runs` +- the only allowed mutation of an existing row is the in-place transition from `running` to a terminal status +- both manual `fetch` executions and future scheduler-triggered executions contribute to the same append-only fetch history + ### 6. `source_records` + Optional normalized raw records before metric extraction. Suggested fields: + - `id` - `source_id` - `binding_id` @@ -179,10 +235,11 @@ Suggested fields: - `dedupe_hash` Uniqueness guidance: + - unique on `(source_id, external_id)` when the upstream source exposes a stable ID - otherwise unique on `dedupe_hash` -`source_records` may be useful later, but it is **not** part of the minimum `v0.3.0` contract freeze. The first live ingestion baseline may write directly to `metric_snapshots` and `content_items` without introducing this intermediate table. +`source_records` may be useful later, but it is **not** part of the minimum `v0.4.0` contract freeze. The current ingestion baseline can continue to write directly to `metric_snapshots` and `content_items` without introducing this intermediate table. ### 7. `metric_snapshots` @@ -230,6 +287,12 @@ Important implementation notes: - `news.article_count` uses `unit = count` and `window_key = point_in_time` - owner-scoped `github.contributors` is intentionally deferred +Repeated-run semantics frozen for `v0.4.0`: + +- snapshots remain append-only across successful reruns +- a rerun that observes the same numeric or text value still persists a new row when `captured_at` is new +- later processing must read ordered history rather than overwrite a synthetic latest-metric row in place + ### 8. `content_items` Deduplicated feed items shown in later unified-feed style views. @@ -271,10 +334,54 @@ Implemented dedupe behavior: 4. first-write values remain authoritative for `discovered_at`, title, summary, URL, and metadata 5. cross-binding reconciliation remains deferred -### 9. `alert_rules` -Runtime rules loaded from configuration and optionally materialized in storage later. +Repeated-run semantics frozen for `v0.4.0`: + +- content dedupe is rerun-safe, not history-destructive +- previously seen dedupe hits are ignored without mutating first-write values +- newly discovered content appends new rows +- later alert and candidate processing must count distinct persisted evidence, not processor passes + +### 9. `schedule_runs` + +Job-level execution history for one scheduled source refresh. + +Frozen `v0.4.0` conceptual fields: + +- `id` +- `dataset_id` +- `schedule_job_id` +- `source_id` +- `status` +- `started_at` +- `finished_at` +- `error_message` +- `fetch_runs_started` +- `fetch_runs_succeeded` +- `fetch_runs_failed` + +Frozen lifecycle: + +| Status | Terminal | Required timestamps | Semantics | +| --- | --- | --- | --- | +| `running` | no | `started_at` required, `finished_at` must be null | the due job execution has started | +| `succeeded` | yes | `started_at` and `finished_at` required | all downstream fetch work completed without failures | +| `partial` | yes | `started_at` and `finished_at` required | at least one downstream fetch succeeded and at least one failed | +| `failed` | yes | `started_at` and `finished_at` required | the job reached a terminal failure without any successful downstream fetch work | + +Frozen execution semantics: + +- one row represents one due execution of one `schedule_jobs` row +- not-due checks do not persist synthetic `schedule_runs` rows +- due calculation should be anchored on the most recent terminal `schedule_runs.started_at` plus `schedule_jobs.cadence` +- `schedule_runs` summarize job-level orchestration, while per-binding detail remains in `fetch_runs` +- repeated scheduler executions append new rows rather than replace prior scheduler history + +### 10. `alert_rules` + +Runtime rules loaded from configuration and persisted as catalog/config state. + +Current implemented catalog/config fields include: -Suggested fields: - `id` - `name` - `source_id` @@ -284,11 +391,26 @@ Suggested fields: - `severity` - `enabled` -### 10. `alert_events` -Materialized alert instances raised by processors. +Frozen `v0.4.0` config-to-persistence mapping: + +- metric-based rules evaluate persisted `metric_snapshots` +- `event_key = "release.published"` evaluates persisted `content_items` with `item_type = "release"` +- `event_key = "news.co_mention"` evaluates repeated co-mention evidence derived from persisted `content_items` with `item_type = "news_article"` +- `window` defines the comparison or repetition lookback interval +- `condition.value` or `condition.silence_days` defines the trigger threshold +- `cooldown` suppresses duplicate `alert_events`, but does not erase prior alert history +- rules with `condition.type = "candidate_repeat_gte"` may also materialize `candidate_suggestions` + +No additional config fields are required for the initial `v0.4.0` processing baseline. + +### 11. `alert_events` + +Materialized alert instances raised by processors over persisted observations. + +Frozen `v0.4.0` conceptual fields: -Suggested fields: - `id` +- `dataset_id` - `alert_rule_id` - `entity_id` - `source_id` @@ -296,14 +418,34 @@ Suggested fields: - `status` - `summary` - `payload_json` +- `observed_at` - `triggered_at` - `resolved_at` -### 11. `candidate_entities` -Potential new competitors or entities discovered from repeated co-mentions. +Expected statuses: + +- `open` +- `resolved` + +Frozen persistence semantics: + +- `alert_rules.id` maps to `alert_events.alert_rule_id` +- `alert_rules.source_id` maps to `alert_events.source_id` +- `alert_rules.severity` maps to `alert_events.severity` +- rule-specific evidence belongs in `payload_json` +- reevaluating unchanged persisted evidence must not create duplicate open rows for the same logical alert instance +- resolving a still-open condition updates the existing row to `resolved` and sets `resolved_at` +- a later distinct recurrence may create a new alert row after cooldown/resolution conditions are satisfied + +### 12. `candidate_suggestions` + +Reviewer-facing suggestions for untracked entities discovered from repeated persisted evidence. + +Frozen `v0.4.0` conceptual fields: -Suggested fields: - `id` +- `dataset_id` +- `alert_rule_id` - `candidate_key` - `display_name` - `discovered_from_entity_id` @@ -315,43 +457,105 @@ Suggested fields: - `evidence_json` Expected statuses: + - `pending_review` - `accepted` - `dismissed` -## Timestamp semantics frozen for `v0.3.0` +Frozen creation/update semantics: + +- a row is created when a rule with `event_key = "news.co_mention"` and `condition.type = "candidate_repeat_gte"` first crosses threshold +- uniqueness should be anchored on `(dataset_id, discovered_from_entity_id, source_id, candidate_key)` +- rerunning processing without new distinct evidence must not create duplicate rows or increase `mention_count` +- later new evidence should update `mention_count`, `last_seen_at`, and `evidence_json` while preserving the same canonical suggestion row +- review status changes must not discard evidence history +- accepting a suggestion does **not** imply automatic tracked-entity creation in `v0.4.0` + +## Timestamp semantics frozen for `v0.4.0` + All timestamps should be stored in RFC3339 UTC format. ### `started_at` + - required on every `fetch_runs` row - marks when a binding execution begins ### `finished_at` + - required when `fetch_runs.status` is `succeeded` or `failed` - must remain null while the row is `running` - marks when the binding execution reaches a terminal outcome ### `captured_at` + - required on every `metric_snapshots` row - represents the observation time of the upstream state for that successful binding execution - should be assigned once per successful binding execution and reused across the metric rows emitted by that execution - should satisfy `started_at <= captured_at <= finished_at` ### `published_at` + - source-provided publication timestamp for a `content_item` - may be null when the upstream source does not expose a trustworthy publication time - must not be synthesized from `captured_at` ### `discovered_at` + - first successful observation time for a `content_item` -- should be set to the successful run's first-seen observation time when the row is inserted +- should be set to the successful run’s first-seen observation time when the row is inserted - must never be updated on rerun dedupe hits +### `schedule_runs.started_at` + +- required on every `schedule_runs` row +- marks when a due scheduled job actually begins execution +- anchors the next due calculation for that job + +### `schedule_runs.finished_at` + +- required when `schedule_runs.status` is `succeeded`, `partial`, or `failed` +- must remain null while the row is `running` +- marks when the scheduled job reaches a terminal outcome + +### `alert_events.observed_at` + +- required on every `alert_events` row +- represents when the triggering evidence was observed in persisted history +- for metric rules, this should usually come from the triggering snapshot’s `captured_at` +- for release/activity rules, this should prefer the content event’s trustworthy upstream `published_at` when available, otherwise the persisted `discovered_at` + +### `alert_events.triggered_at` + +- required on every `alert_events` row +- marks when the processor materializes the alert row in SQLite +- should satisfy `observed_at <= triggered_at` + +### `alert_events.resolved_at` + +- null while an alert is still open +- set when a previously open alert is marked `resolved` +- must be greater than or equal to `triggered_at` + +### `candidate_suggestions.first_seen_at` + +- required on every `candidate_suggestions` row +- marks the first time the system observed distinct evidence for that suggestion +- should be derived from persisted evidence discovery time, usually `content_items.discovered_at` + +### `candidate_suggestions.last_seen_at` + +- required on every `candidate_suggestions` row +- marks the latest time the system observed distinct evidence for that suggestion +- must not advance on processor reruns that do not introduce new evidence + ## Read-side models + The UI and API should consume read models derived from the write-side tables above. ### Entity overview row + Suggested fields: + - `entity_id` - `name` - `kind` @@ -361,13 +565,17 @@ Suggested fields: - latest activity timestamp ### Entity metrics timeline + Suggested fields: + - `entity_id` - `metric_key` - time-series points ### Head-to-head comparison row + Suggested fields: + - `left_entity_id` - `right_entity_id` - `metric_key` @@ -377,7 +585,9 @@ Suggested fields: - `right_delta_window_value` ### Unified feed item + Suggested fields: + - `entity_id` - `entity_name` - `item_type` @@ -387,7 +597,9 @@ Suggested fields: - `source_id` ### Alert feed item + Suggested fields: + - `alert_event_id` - `entity_id` - `severity` @@ -395,10 +607,10 @@ Suggested fields: - `triggered_at` - `status` -## Core data contracts -The following normalized contracts are frozen conceptually for `v0.3.0` even though the concrete Go structs and SQL migrations still land in later issues. +## Core data contract examples ### `FetchRun` + ```json { "source_id": "github", @@ -414,21 +626,23 @@ The following normalized contracts are frozen conceptually for `v0.3.0` even tho ``` ### `MetricSnapshot` + ```json { "entity_id": "project-gdk", "source_id": "github", "binding_id": "github-project-gdk", - "metric_key": "github.commits", + "metric_key": "github.commits_30d", "metric_value_num": 42, "unit": "count", - "window_key": "trailing_30d", + "window_key": "point_in_time", "scope_key": "repo:Blockstream/gdk", "captured_at": "2026-04-12T00:00:01Z" } ``` ### `ContentItem` + ```json { "entity_id": "product-jade", @@ -444,9 +658,57 @@ The following normalized contracts are frozen conceptually for `v0.3.0` even tho } ``` +### `ScheduleRun` + +```json +{ + "schedule_job_id": "news-hourly", + "source_id": "news-rss", + "status": "partial", + "started_at": "2026-04-17T12:00:00Z", + "finished_at": "2026-04-17T12:00:20Z", + "fetch_runs_started": 18, + "fetch_runs_succeeded": 17, + "fetch_runs_failed": 1, + "error_message": "" +} +``` + +### `AlertEvent` + +```json +{ + "alert_rule_id": "github-stars-7d-spike", + "entity_id": "org-blockstream", + "source_id": "github", + "severity": "medium", + "status": "open", + "summary": "GitHub stars increased by at least 20 percent in 7 days", + "observed_at": "2026-04-17T12:00:01Z", + "triggered_at": "2026-04-17T12:00:05Z", + "resolved_at": null +} +``` + +### `CandidateSuggestion` + +```json +{ + "alert_rule_id": "news-candidate-co-mention", + "candidate_key": "company:trezor", + "display_name": "Trezor", + "discovered_from_entity_id": "product-jade", + "source_id": "news-rss", + "mention_count": 3, + "first_seen_at": "2026-04-03T09:00:00Z", + "last_seen_at": "2026-04-17T12:00:01Z", + "status": "pending_review" +} +``` + ## SQLite baseline outline -The current SQLite schema now includes both catalog and ingestion tables. +The current SQLite schema includes both catalog tables and first-class ingestion observations. | Table | Purpose | Status | | --- | --- | --- | @@ -461,25 +723,60 @@ The current SQLite schema now includes both catalog and ingestion tables. | `metric_snapshots` | append-only historical metric observations | implemented | | `content_items` | deduplicated feed content | implemented | | `source_records` | optional raw staging | deferred | -| `alert_events` | materialized alerts | deferred | -| `candidate_entities` | discovered competitor candidates | deferred | +| `schedule_runs` | scheduled job execution history | contract frozen for `v0.4.0`, not implemented yet | +| `alert_events` | materialized alerts | contract frozen for `v0.4.0`, not implemented yet | +| `candidate_suggestions` | discovered competitor suggestions | contract frozen for `v0.4.0`, not implemented yet | + +## Reviewer-visible SQLite inspection path frozen for `v0.4.0` + +### Current implemented inspection path + +Today, reviewers can already inspect the implemented ingestion tables directly in SQLite: + +```bash +sqlite3 /tmp/signalscope.db "SELECT id, source_id, binding_id, status, started_at, finished_at, records_written, metrics_written, content_items_written FROM fetch_runs ORDER BY id DESC LIMIT 20;" +sqlite3 /tmp/signalscope.db "SELECT binding_id, metric_key, metric_value_num, metric_value_text, scope_key, captured_at FROM metric_snapshots ORDER BY id DESC LIMIT 20;" +sqlite3 /tmp/signalscope.db "SELECT binding_id, item_type, title, url, published_at, discovered_at FROM content_items ORDER BY id DESC LIMIT 20;" +``` + +### Planned `v0.4.0` inspection path after backend issues land + +Once the corresponding scheduler and processing issues are implemented, the intended reviewer-visible SQLite path is: + +```bash +sqlite3 /tmp/signalscope.db "SELECT id, schedule_job_id, source_id, status, fetch_runs_started, fetch_runs_succeeded, fetch_runs_failed, started_at, finished_at FROM schedule_runs ORDER BY id DESC LIMIT 20;" +sqlite3 /tmp/signalscope.db "SELECT id, alert_rule_id, entity_id, source_id, severity, status, observed_at, triggered_at, resolved_at FROM alert_events ORDER BY id DESC LIMIT 20;" +sqlite3 /tmp/signalscope.db "SELECT id, alert_rule_id, candidate_key, display_name, discovered_from_entity_id, source_id, mention_count, first_seen_at, last_seen_at, status FROM candidate_suggestions ORDER BY id DESC LIMIT 20;" +``` + +Important honesty note: + +- the queries for `schedule_runs`, `alert_events`, and `candidate_suggestions` are part of the committed `v0.4.0` contract freeze +- they are **not** expected to return rows until the corresponding implementation issues land +- the current CLI still needs backend changes before repeated same-DB reviewer history is preserved automatically ## Lifecycle from ingestion to dashboard + 1. configuration defines entities, sources, schedules, and alert rules -2. a fetcher runs against one source binding -3. the binding execution is recorded in `fetch_runs` -4. metrics are written to `metric_snapshots` -5. feed-like content is written to `content_items` -6. later processors compare recent observations and may raise alert events -7. API and dashboard read models are assembled from persisted data +2. a manual fetch or a due scheduled job begins +3. when the scheduler path is implemented, the due job execution is recorded in `schedule_runs` +4. each binding execution is recorded in `fetch_runs` +5. metrics are written to `metric_snapshots` +6. feed-like content is written to `content_items` +7. later processors compare persisted history and evaluate alert rules +8. processors may materialize `alert_events` and `candidate_suggestions` +9. API and dashboard read models are assembled from persisted data later ## Deferred implementation details -The exact SQL DDL, migrations, indexes, and Go structs remain deferred to implementation issues. -What is frozen here is the conceptual model and the minimum set of persisted observation contracts required for a trustworthy `v0.3.0` ingestion path. +The exact SQL DDL, migrations, indexes, and Go structs remain deferred to later implementation issues. + +What is frozen here is the conceptual model and the minimum set of persisted observation contracts required for a trustworthy `v0.4.0` backend-processing path. Also intentionally deferred beyond this contract freeze: + +- exact scheduler CLI/runtime implementation details - cross-binding content-item dedupe - optional `source_records` staging details -- alert and candidate runtime materialization -- scheduler, API, and dashboard behavior +- automatic tracked-entity creation from accepted candidate suggestions +- API and dashboard behavior diff --git a/docs/roadmap.md b/docs/roadmap.md index 8b710c2..1cb0e36 100644 --- a/docs/roadmap.md +++ b/docs/roadmap.md @@ -4,7 +4,7 @@ This roadmap was reviewed and reconciled for the implemented `v0.3.0` ingestion baseline. -The repository is no longer at a catalog/bootstrap-only stage. It now has a real fetch runtime, persisted observations, and deterministic test coverage for ingestion. The roadmap below therefore treats `v0.3.0` as the current baseline and keeps later milestones focused on what is still actually missing. +The repository is no longer at a catalog/bootstrap-only stage. It now has a real fetch runtime, persisted observations, and deterministic test coverage for ingestion. The roadmap below therefore treats `v0.3.0` as the current runtime baseline and `v0.4.0` as the next committed backend-only milestone. ## Delivery strategy @@ -43,7 +43,7 @@ That means: - repeated `fetch` commands against the same SQLite path replace earlier ingestion rows for that dataset - reviewers should inspect the database after each run or use separate SQLite paths per source kind -- changing that runtime behavior is a follow-up concern, not part of the current documentation issue +- changing that runtime behavior is the committed `v0.4.0` backend milestone, not later API or dashboard work ## Milestone plan @@ -53,22 +53,21 @@ That means: | `v0.2.0` | runtime foundation | config loader, validation, SQLite schema, migrations, seed-demo flow | Released | | `v0.2.1` | runtime hardening | docs reconciliation, contract enforcement, fixture normalization, repo hygiene | Released | | `v0.3.0` | source ingestion | live fetch runtime, GitHub/news/changelog fetchers, persisted observations, fetch-run tracking | Current baseline | -| `v0.4.0` | processing and alerts | scheduler runtime, delta detection, alert generation, candidate discovery baseline | Planned | +| `v0.4.0` | history-preserving processing and alerts | append-only repeated-run history, scheduler execution, derived alert/candidate persistence | Planned | | `v0.5.0` | delivery surface | read-only API, entity views, compare page, unified feed, alert feed | Planned | | `v0.6.0` | polish and release | cleanup, reviewer-flow verification, reporting polish, release packaging | Planned | ## Next implementation order after `v0.3.0` -### 1. Add processing on top of persisted observations (`v0.4.0`) +### 1. Add history-preserving processing on top of persisted observations (`v0.4.0`) The next milestone should build on the now-persisted write side: -- scheduled execution over configured jobs -- delta detection over recent metric/content history -- alert materialization -- candidate competitor discovery - -These features should continue to read persisted SQLite state rather than call upstream APIs directly. +- preserve repeated same-DB `fetch_runs`, `metric_snapshots`, and deduplicated `content_items` history rather than replacing earlier rows +- execute due `schedule_jobs` through a deterministic one-shot scheduler with persisted `schedule_runs` +- evaluate persisted observations into materialized `alert_events` +- surface repeated untracked co-mentions as persisted `candidate_suggestions` +- keep API and dashboard work deferred until `v0.5.0` ### 2. Add delivery surfaces after read models stabilize (`v0.5.0`) @@ -96,8 +95,8 @@ Later cleanup can focus on: | --- | --- | --- | | GitHub or feed rate limits during manual review | reviewer confusion or partial fetch coverage | document optional `GITHUB_TOKEN`, keep CI fixture-based, record failures in `fetch_runs` | | drifting documentation after the runtime became real | trust erosion | update docs in the same PR whenever runtime truth changes | -| over-scoping scheduler/API/dashboard work too early | milestone slippage | keep `v0.4.0` about processing and `v0.5.0` about delivery | -| assuming multi-source accumulation from the current CLI | misleading reviewer instructions | document the current reseed-on-fetch behavior explicitly | +| over-scoping scheduler/API/dashboard work too early | milestone slippage | keep `v0.4.0` about backend processing and `v0.5.0` about delivery | +| assuming multi-source accumulation from the current CLI | misleading reviewer instructions | document the current reseed-on-fetch behavior explicitly and freeze the intended `v0.4.0` history-preserving target honestly | | hiding partial failures behind aggregate success text | difficult debugging | preserve per-binding `fetch_runs` and non-zero CLI status on any failed binding | ## Definition of done for the `v0.3.0` baseline @@ -116,6 +115,7 @@ A reviewer should be able to: The roadmap now reflects a repository that already crossed the line from “foundation only” to “real ingestion baseline.” The next work should not re-document `v0.3.0`; it should build on it: -- `v0.4.0` for processing + +- `v0.4.0` for history-preserving backend processing - `v0.5.0` for delivery - `v0.6.0` for polish and release packaging