Skip to content

feat: record and reload scan provenance - #105

Open
YalcinkayaE wants to merge 2 commits into
AI-Safety-Institute:mainfrom
YalcinkayaE:feat/run-provenance
Open

YalcinkayaE wants to merge 2 commits into
AI-Safety-Institute:mainfrom
YalcinkayaE:feat/run-provenance

Conversation

@YalcinkayaE

@YalcinkayaE YalcinkayaE commented Oct 29, 2026 •

Copy link
Copy Markdown

Part of #20. Independent of #117.

Record Transect's added run identities in Scout's scan metadata: installed Transect version, canonical spec_id, Spec source path and bytes hash, sample / epochs arguments, selected eval-log file hashes, and custom layer source identities.

TransectResults.provenance is a read-time view assembled by load(). Its scan field exposes Scout's recorded header and each scanner's own parameters. The combined view is not stored again. OpenClaw uses Scout's recorded snapshot location without hashing database files. Scans without the added metadata return provenance=None.

The revision removes the extra Spec copies, model serialisation, duplicate judge validation and scan-construction reorder. Documentation is reduced to the existing load discussion, an architecture row and a short skill pointer.

Rebased onto current main and bumped to 0.7.1. The lockfile diff contains only Transect's package-version line; dependency pins, platform markers and the seven-day cooldown are unchanged. uv lock --check and uv sync --locked pass with CI's pinned uv 0.13.5.

Validation

  • make check: 672 passed, 10 skipped; Ruff, formatting, mypy and Pyright pass.
  • Final targeted provenance and skill checks: 13 passed, with type checks passing after the custom-factory source correction.
  • Scan/reload coverage checks metadata, scanner parameters, source hashes, absence of database hashing, reload after source removal and legacy stores. Canonical Spec identity is tested separately.
  • Wheel and sdist build; an isolated wheel import exposes RunProvenance and version 0.7.1.

AI assistance: implementation and tests prepared with Codex. Tests use mock judges or structural scans and make no paid model calls.

@YalcinkayaE

Copy link
Copy Markdown
Author

A focused first slice is ready for review. It covers canonical Spec identity, effective run and judge settings, input/custom-layer identities, and reload of the record. It deliberately leaves the transcripts_location replay override in #20 as a follow-up.

Local make check passes with 560 tests and 7 existing skips; the GitHub CI run is currently action_required because this is a first-time fork contribution and awaits workflow approval.

@alexandraabbas alexandraabbas left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Review posted on behalf of @alexandraabbas (written with Claude Code).

Thanks for taking on #20. Before going further we want to narrow this to what is genuinely new, because most of what RunProvenance records is already in the scan store, and we do not want the same fact in three or four places that can drift.

What _scan.json already records today (checked on tests/fixtures/demo_scan):

  • The validated spec as a scanner param on decision_phases and subagent_classification.
  • judge_models in exactly the shape model_identity() produces (Scout serialises a Model param to model, config, base_url, model_args itself), plus k_rolls, chunk, digest, verify, verify_chunk, verify_sample, verifier_model, narrate, activity.
  • Scout's own header: scan_id, timestamp, revision (git origin, commit, version), packages (inspect_ai, inspect_scout), options (max_transcripts, max_processes), and transcripts.transcript_ids mapping each id to its source path.
  • Every judged result carries the judge block (regime, roster, k_rolls, verifier flags) that the reliability audit reads.

So spec, spec_sha256, most of settings, model_identity(), and the inspect package versions duplicate stored data. With spec_text added, the spec sits in the store four times.

Where each kind of provenance belongs

  • Scan-level run provenance (when, which code, which inputs, which Scout options) is Scout's header. Scout's scan() also takes metadata: dict and tags, and ScanSpec.metadata is readable on load. That is the right carrier for anything transect adds at run level, not a provenance= argument smuggled through the eval_setup scanner's params.
  • Judge provenance (models, rolls, verifier, digest, chunking, the spec the judge was given) belongs to the judged scanner: its params plus the per-result judge block. A custom judged layer carries its own, so a run-level copy would be wrong for a run with several judged surfaces.
  • What no scanner owns is the real gap, and it is what this PR should record: the transect package version (Scout's revision.version is a git describe of the cwd, not the installed version), the spec source path, bytes hash and spec_id, the sample / epochs selection arguments, input file hashes, and custom layer source identities.

Concrete asks

  1. Record only the items in the last bullet, via Scout's metadata= on scan(), and read them back from results.spec.metadata in load(). Drop spec, spec_text, spec_sha256, packages other than transect, and every settings key a scanner already records.
  2. TransectResults.provenance can still present one object, but as a read-time view that pulls the judge facts from the scanner params and the run facts from Scout's header and metadata, not as a second stored copy.
  3. Drop model_identity(); Scout already does that serialisation.
  4. Drop _validate_judge_setup() and the reorder of the OpenClaw import and selection block. The empty-list check duplicates the one in _scanners(), and the reorder is only there because the provenance has to be built before the scanners are constructed. With metadata= on scan() the hashes can be computed after selection without moving anything.
  5. Input hashing reads every input file in full on every run. That is fine for a demo log and expensive for multi-GB OpenClaw snapshot databases. Hash eval-log inputs, and for OpenClaw record the snapshot directory the scan reads rather than hashing it. The current input_hashes(transcripts: Any) duck-types across two shapes; after the narrowing it should take list[TranscriptInfo] and nothing else.
  6. Docs shrink with the scope: no new "Provenance and replay" README section, no new load-bearing contract bullet in AGENTS.md. One paragraph under the existing load() discussion and the architecture-table row are enough. The README sentence "the scan store now retains the full family Spec and resolved input paths" describes what Scout already did before this PR.
  7. Rebase and re-bump: 0.6.0 to 0.6.1 is stale, main is at 0.7.0. The uv.lock diff also carries a lock revision change and dropped platform markers, which means it was re-locked with a different uv than CI pins (UV_VERSION in ci.yml); regenerate with the pinned version so only the package version changes.
  8. Tests: the current five are fine in shape. After narrowing, one scan-and-reload test plus the canonical-spec-identity test should be enough.

Happy to look again once it is cut down to the novel items.

@YalcinkayaE

Copy link
Copy Markdown
Author

The head moved after Alexandra Abbas's review at 10:05 Europe/London. Current head e202844 was committed at 10:53:27 Europe/London on 5 November 2026.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants