Repository navigation
feat: record and reload scan provenance - #105
YalcinkayaE wants to merge 2 commits into
Conversation
|
A focused first slice is ready for review. It covers canonical Spec identity, effective run and judge settings, input/custom-layer identities, and reload of the record. It deliberately leaves the Local |
6ed7d05 to
9838cf8
Compare
alexandraabbas
left a comment
There was a problem hiding this comment.
Review posted on behalf of @alexandraabbas (written with Claude Code).
Thanks for taking on #20. Before going further we want to narrow this to what is genuinely new, because most of what RunProvenance records is already in the scan store, and we do not want the same fact in three or four places that can drift.
What _scan.json already records today (checked on tests/fixtures/demo_scan):
- The validated
specas a scanner param ondecision_phasesandsubagent_classification. judge_modelsin exactly the shapemodel_identity()produces (Scout serialises aModelparam to model, config, base_url, model_args itself), plusk_rolls,chunk,digest,verify,verify_chunk,verify_sample,verifier_model,narrate,activity.- Scout's own header:
scan_id,timestamp,revision(git origin, commit, version),packages(inspect_ai, inspect_scout),options(max_transcripts, max_processes), andtranscripts.transcript_idsmapping each id to its source path. - Every judged result carries the
judgeblock (regime, roster, k_rolls, verifier flags) that the reliability audit reads.
So spec, spec_sha256, most of settings, model_identity(), and the inspect package versions duplicate stored data. With spec_text added, the spec sits in the store four times.
Where each kind of provenance belongs
- Scan-level run provenance (when, which code, which inputs, which Scout options) is Scout's header. Scout's
scan()also takesmetadata: dictandtags, andScanSpec.metadatais readable on load. That is the right carrier for anything transect adds at run level, not aprovenance=argument smuggled through theeval_setupscanner's params. - Judge provenance (models, rolls, verifier, digest, chunking, the spec the judge was given) belongs to the judged scanner: its params plus the per-result
judgeblock. A custom judged layer carries its own, so a run-level copy would be wrong for a run with several judged surfaces. - What no scanner owns is the real gap, and it is what this PR should record: the transect package version (Scout's
revision.versionis a git describe of the cwd, not the installed version), the spec source path, bytes hash andspec_id, thesample/epochsselection arguments, input file hashes, and custom layer source identities.
Concrete asks
- Record only the items in the last bullet, via Scout's
metadata=onscan(), and read them back fromresults.spec.metadatainload(). Dropspec,spec_text,spec_sha256,packagesother than transect, and everysettingskey a scanner already records. TransectResults.provenancecan still present one object, but as a read-time view that pulls the judge facts from the scanner params and the run facts from Scout's header and metadata, not as a second stored copy.- Drop
model_identity(); Scout already does that serialisation. - Drop
_validate_judge_setup()and the reorder of the OpenClaw import and selection block. The empty-list check duplicates the one in_scanners(), and the reorder is only there because the provenance has to be built before the scanners are constructed. Withmetadata=onscan()the hashes can be computed after selection without moving anything. - Input hashing reads every input file in full on every run. That is fine for a demo log and expensive for multi-GB OpenClaw snapshot databases. Hash eval-log inputs, and for OpenClaw record the snapshot directory the scan reads rather than hashing it. The current
input_hashes(transcripts: Any)duck-types across two shapes; after the narrowing it should takelist[TranscriptInfo]and nothing else. - Docs shrink with the scope: no new "Provenance and replay" README section, no new load-bearing contract bullet in AGENTS.md. One paragraph under the existing
load()discussion and the architecture-table row are enough. The README sentence "the scan store now retains the full family Spec and resolved input paths" describes what Scout already did before this PR. - Rebase and re-bump: 0.6.0 to 0.6.1 is stale,
mainis at 0.7.0. Theuv.lockdiff also carries a lockrevisionchange and dropped platform markers, which means it was re-locked with a different uv than CI pins (UV_VERSIONinci.yml); regenerate with the pinned version so only the package version changes. - Tests: the current five are fine in shape. After narrowing, one scan-and-reload test plus the canonical-spec-identity test should be enough.
Happy to look again once it is cut down to the novel items.
9838cf8 to
e202844
Compare
|
The head moved after Alexandra Abbas's review at 10:05 Europe/London. Current head e202844 was committed at 10:53:27 Europe/London on 5 November 2026. |
Part of #20. Independent of #117.
Record Transect's added run identities in Scout's scan metadata: installed Transect version, canonical
spec_id, Spec source path and bytes hash,sample/epochsarguments, selected eval-log file hashes, and custom layer source identities.TransectResults.provenanceis a read-time view assembled byload(). Itsscanfield exposes Scout's recorded header and each scanner's own parameters. The combined view is not stored again. OpenClaw uses Scout's recorded snapshot location without hashing database files. Scans without the added metadata returnprovenance=None.The revision removes the extra Spec copies, model serialisation, duplicate judge validation and scan-construction reorder. Documentation is reduced to the existing load discussion, an architecture row and a short skill pointer.
Rebased onto current main and bumped to 0.7.1. The lockfile diff contains only Transect's package-version line; dependency pins, platform markers and the seven-day cooldown are unchanged.
uv lock --checkanduv sync --lockedpass with CI's pinned uv 0.13.5.Validation
make check: 672 passed, 10 skipped; Ruff, formatting, mypy and Pyright pass.RunProvenanceand version 0.7.1.AI assistance: implementation and tests prepared with Codex. Tests use mock judges or structural scans and make no paid model calls.