Source resolution and deep link provenance (v0.2.0)

CHANGELOG.md

@@ -6,12 +6,31 @@ All notable changes to RESYNTH are documented here. The format follows
 
 ## [Unreleased]
 
-### Changed
-- Default model for the Claude Code operator is now Fable 5
-  (`claude-fable-5`), Anthropic's newest and fastest top-tier model;
-  override per workspace in `operator.yaml` as before.
+## [0.2.0] - 2026-06-12
 
 ### Added
+- Source resolution (`resynth resolve`): follows links and file references
+  found inside ingested sources and registers what it fetches as new first
+  class sources with provenance. Five target kinds: html articles, pdf
+  links, local files, YouTube videos and Vimeo videos. Public video
+  captions become timestamped transcripts.
+- Transcript pending stubs: a video without public captions still becomes
+  a real source, and a later successful fetch upgrades the stub in place,
+  keeping its source id.
+- The resolution manifest at `index/resolution.jsonl`: records every
+  target and its outcome so re-runs are idempotent. Fetched and duplicate
+  targets are never retried, failed and pending ones are.
+- Schema v2 source frontmatter: `source_type`, `url`, `resolved_from` and,
+  for video sources, `transcript_status`.
+- Optional `source_locator` on claims, a deep link into the source built
+  from a url, page, timestamp or anchor, validated by `extract-verify`.
+- `resynth migrate`: explicit upgrade of a project's sources to schema v2.
+  Bodies and content hashes are untouched, re-sealing stays a separate
+  operator step.
+- `resynth --version`.
+- MASTER.json format `resynth-master/2` with a sources array, plus a
+  `load_master` reader that accepts both `/1` and `/2`.
+- The guided wizard offers source resolution straight after intake.
 - Completion ping: when a delegated AI step runs longer than 90 seconds,
   RESYNTH plays a sound and shows a desktop notification (Windows toast /
   macOS notification) when it finishes, and again when the master document
@@ -21,6 +40,13 @@ All notable changes to RESYNTH are documented here. The format follows
   the assistant's output streamed as it arrives, instead of a silent prompt
   until completion.
 
+### Changed
+- Default model for the Claude Code operator is now Fable 5
+  (`claude-fable-5`), Anthropic's newest and fastest top-tier model;
+  override per workspace in `operator.yaml` as before.
+- The MASTER.md source register gains Type and Link columns, so every
+  source's kind and origin url are visible in the sealed master.
+
 ### Fixed
 - Sealing failed with "paths are ignored by one of your .gitignore files"
   in workspaces where `projects/*` is gitignored (any workspace cloned from

DECISIONS.md

@@ -27,3 +27,9 @@ Every architectural decision with a one line rationale.
 - The wired assistant defaults to Claude Code with claude-opus-4-8 at high reasoning effort, adjustable with resynth operator.
 - Delegated operator steps verify against the stage gate and retry up to three times with the gate reasons fed back, then fall back to manual mode.
 - The brief step asks whether reports already exist and skips prompt generation when they do, consolidation only is a first class flow.
+- resolve is a stage 01 verb that re-evaluates the intake gate, not a sixth gate, so already sealed projects never grow a phantom PENDING gate.
+- Fetchers use only the Python standard library (urllib, html.parser, xml.etree), so the four dependency decision holds.
+- Video transcripts are best effort, a pending stub is a real source that upgrades in place and keeps its source id, so claim ids built on it stay stable.
+- Source schema v2 is adopted only through an explicit resynth migrate, never silently, and re-sealing afterwards stays an operator act.
+- MASTER.json format resynth-master/2 ships together with a load_master reader that accepts both /1 and /2, so downstream consumers never break on old exports.
+- Resolution is depth one by design, fetched sources are not scanned for further links unless the operator forces a re-scan with --source.

README.md

@@ -78,11 +78,47 @@ agent then drives the pipeline below, and the result is one BEST master
 document, readable by humans, verifiable by machine, and exportable as JSON
 for a downstream AI agent to action.
 
+After intake there is one optional extra step. RESYNTH offers to fetch the
+links and file references found inside your reports and register them as
+extra sources, so the things your reports cite become evidence too.
+
 ```
 chat -> brief -> per platform prompts -> research reports -> intake ->
-extract -> reconcile -> synthesise -> audit -> seal -> MASTER.md + MASTER.json
+resolve (optional) -> extract -> reconcile -> synthesise -> audit -> seal ->
+MASTER.md + MASTER.json
+```
+
+## Fetching linked sources
+
+Research reports cite things. `resynth resolve <project>` follows those
+citations and turns them into first class sources of their own. It scans
+every ingested report for links and file references, fetches each one, and
+registers the result with provenance back to the report that mentioned it.
+It handles html articles, pdf links, local files, and YouTube and Vimeo
+videos. Public video captions become timestamped transcripts.
+
+When a video has no public captions, resolve still creates the source as a
+pending transcript stub. You can paste a transcript into the stub yourself,
+or re-run resolve later to retry. A later successful fetch upgrades the stub
+in place and keeps the same source id.
+
+Resolution is idempotent. Every outcome is recorded in
+`index/resolution.jsonl`, fetched and duplicate targets are never fetched
+twice, and failed or pending targets are retried on the next run. Fetching
+is polite: robots.txt is honoured, requests to the same host are spaced one
+second apart, and responses are capped at 10 MiB. Resolution goes one level
+deep only. Fetched sources are not scanned for further links unless you
+force a re-scan with `--source`.
+
+```
+resynth resolve myproject
+resynth resolve myproject --only youtube    # just targets matching a substring
+resynth resolve myproject --source S03      # re-scan one source, even a fetched one
 ```
 
+The full reference, including the manifest format, the source schema and
+the migration guide, lives in [docs/SOURCE-RESOLUTION.md](docs/SOURCE-RESOLUTION.md).
+
 ## Install
 
 ```
@@ -135,6 +171,7 @@ scripts/run_demo.py and in the end to end test.
 resynth init <project>            create project skeleton plus default merge-rules.yaml
 resynth brief <project> --topic   capture the research question, generate the prompt workspace
 resynth intake <project> --source <file> ...   stage 1, repeatable per file
+resynth resolve <project>         fetch links and file references inside sources as new first class sources
 resynth extract <project>         stage 2 workspace generation
 resynth extract-verify <project>  stage 2 gate
 resynth reconcile <project>       stage 3, also evaluates the gate
@@ -144,6 +181,7 @@ resynth audit <project>           stage 5 coverage, drift, traceability
 resynth seal <project>            hash everything, commit SEAL.yaml, tag the repo
 resynth export <project>          machine readable output/MASTER.json for agents
 resynth status <project>          gate dashboard
+resynth migrate <project>         upgrade a project's sources to the current schema (v2)
 resynth operator                  show or set the wired AI assistant, model and effort
 resynth doctor                    environment probe
 ```
@@ -173,6 +211,11 @@ Then run: resynth extract-verify <project> --json and fix every violation
 until the gate reports PASS.
 ```
 
+Each claim may also carry an optional `source_locator`, a deep link into the
+source built from a url, a page number, a timestamp or an anchor. The full
+claim and source schemas live in
+[docs/SOURCE-RESOLUTION.md](docs/SOURCE-RESOLUTION.md).
+
 ### Agent prompt for stage 3, reconciliation
 
 ```