Repository navigation
feat(eval): define ADR-0008 x64_freeze.resource_manifest_sha256 digest contract - #26
Merged
Merged
Conversation
…t contract Previously `assert_runtime_boundary` accepted any 64-hex-lowercase string for `x64_freeze.resource_manifest_sha256` with no definition of what bytes it hashes, making the pin unreproducible. Add `raptor.eval.prospective_freeze.compute_resource_manifest_sha256` / `resource_manifest_entries`: a versioned, domain-separated canonical-JSON envelope binding the identity, pinned order, and raw-byte SHA-256 of the three pinned checksum-manifest files (nirvana-grch38-full.sha256.txt, nirvana-grch38-updates.sha256.txt, bias-hg38-data.sha256.txt) already named in configs/eval/core_annotation_bundle.yaml and docs/ops/masked-heldout- bias-rerun-handoff.md. Raw-byte reads make the digest identical on Windows and Linux for byte-identical files, and it changes on any filename/order/ content drift. Add a read-only operator CLI (scripts/compute_adr0008_resource_manifest_sha256.py) for the ADR-0008 x64 worker, a full spec/ops doc (docs/ops/adr-0008-resource-manifest-digest.md), and unit tests. No network access, no BIAS/Nirvana invocation, no mutation of external resource files. assert_runtime_boundary's format-only validation and the existing pre_data_approval schema are unchanged; the draft approval's resource_manifest_sha256 placeholder is still unfilled and still not APPROVED_PRE_DATA. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: 7c146921-f3dd-4a1e-8cf0-8f574de49204
…a gate Governance correction: pre_data_approval (ADR-0020 stages 1-2, ClinVar transport freeze + raw archive acquisition) incorrectly required and validated an x64_freeze block, converting any ADR-0008 runtime-identity drift into a PRE_DATA_DRIFT stop state before any archive GET. ClinVar archive acquisition has no x86-only requirement. - validate_pre_data_approval / execute_transport_and_raw_freeze no longer accept, require, or check x64_freeze / runtime_identity in any form. - New validate_scoring_stage_approval (schema raptor.eval.scoring_stage_approval.v1) is the separate, later, mandatory gate before any BIAS/Nirvana execution or label-dependent evaluation (ADR-0020 stage 4), reusing the existing assert_runtime_boundary / compute_resource_manifest_sha256 contract. All its failures raise ProspectiveInvalidStateError (INVALID / A0 run-integrity), never a PRE_DATA_* stop state, matching the registration spec's own stage-4 rule. - Registration spec YAML gains a scoring_stage_approval section and splits freeze_record_must_pin accordingly; ops doc and operator-script docstrings updated to reflect the new boundary. - pre_data_approval draft approval has its x64_freeze block removed and decision reworded (still not APPROVED_PRE_DATA); a new, separate, unapproved scoring_stage_approval draft file is added with the same placeholder-only resource_manifest_sha256 (no fabricated digest). - Tests updated/added accordingly; no ClinVar archive access performed. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: 7c146921-f3dd-4a1e-8cf0-8f574de49204
Address 4 independent-review findings against the ADR-0008 scoring-stage gate introduced in b40dd08: 1. validate_scoring_stage_approval no longer trusts the approval record's claimed x64_freeze block. It now requires an explicit APPROVED_SCORING_STAGE decision (REJECTED_SCORING_STAGE and any other value are rejected distinctly), an exact approver match against the spec's approver_required, approved_at strictly before an optional first_scoring_execution_at, a separately supplied observed_runtime_identity (worker_designation, worker_arch, bias_commit, nirvana_banner -- never copied from the record), and always independently recomputes resource_manifest_sha256 from resource_manifest_checksums_dir via compute_resource_manifest_sha256. The claimed x64_freeze block must equal that observed+recomputed identity byte-for-byte; a well-formed but fabricated digest (including all-zero) can never pass. 2. adjudicate_prospective_outcomes now calls validate_scoring_stage_approval unconditionally as its first action. No outcome -- including PASS or AUTHORIZED_RESEARCH_ONLY -- can be produced without a verified scoring-stage approval and matching observed runtime identity. 3. The executable acquisition CLI (run_clinvar_2026_08_prospective_freeze.py) no longer accepts --transport-factory. Production/executable transport is hard-wired to prospective_exact_source_transport.build_transport() with no arguments, so no CLI option, config value, or environment variable can substitute a different transport for a confirmed live --execute run. 4. Acquisition (validate_pre_data_approval / execute_transport_and_raw_freeze) no longer verifies scoring_semantics_projection_sha256 or any base eval config hash. That check belongs solely to merge_prospective_overlay, which is no longer called from the acquisition path at all -- only from the (unchanged) scoring-stage config-build surface. Updates the two draft approval JSON files and the registration spec's pre_data_approval / scoring_stage_approval sections to match, and adds targeted positive/negative tests (fabricated/all-zero digest, wrong approver, REJECTED decision, mistimed approval, missing gate at adjudication, alternate live transport, scoring-config drift not blocking acquisition). Validated in WSL2 raptor venv (/home/sdrona/raptor/bin/python, Python 3.12.3): targeted test files pass (194 passed, 3 skipped), and the broader tests/eval suite passes except 2 pre-existing failures in test_tiered_readjudication_frozen_r2.py that are unrelated to this change (file is byte-identical to origin/main). Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: 7c146921-f3dd-4a1e-8cf0-8f574de49204
…scoring boundary (round 4) Additive fixes on top of 323cebe, none amended: 1. Hard-wire confirmed live execution against caller-selected code before the real GET: capture_transport_identity_pin is captured in scripts/run_clinvar_2026_08_prospective_freeze.py immediately after build_transport(), before any lookup-module resolution, and execute_transport_and_raw_freeze re-verifies the pin both on entry and immediately before the network stream, rejecting TRANSPORT_IDENTITY_TAMPERED if a caller-selected lookup callable monkeypatches the transport class mid-call. 2. validate_scoring_stage_approval now takes explicit observation probes (worker designation/arch, BIAS commit, Nirvana banner, resource-manifest checksums dir) instead of trusting a caller- supplied observed_runtime_identity mapping, and independently recomputes resource_manifest_sha256/immutable_inputs_verified rather than accepting the claimed values. 3. adjudicate_prospective_outcomes now requires an immutable first_scoring_execution_at input and rejects PASS/ AUTHORIZED_RESEARCH_ONLY when it is missing, future-dated, or not strictly after approved_at, closing the post-scoring/future- approval gap. 4. Pre-data implementation_freeze validation now also verifies the ACTUALLY EXECUTING module bytes in this process (_implementation_freeze_executing_code_failure), not just the named historical commit's git-show content. 5. immutable_inputs_verified (tsc2.yaml, ACMG/BIAS policy, masking/ predictor-aggregation config, ...) moved out of pre_data_approval and into scoring_stage_approval only; acquisition no longer requires or references any scoring-stage config. 6. Regenerated the pre_data draft's registration.git_blob_sha1/ canonical_lf_sha256 against the current spec bytes, added immutable_inputs_verified + evidence to the scoring_stage draft, and added executing-commit re-pin guidance for final approval. Updated docs/ops/adr-0008-resource-manifest-digest.md and the registration spec YAML to document all six boundary/verification changes; updated all three protected red test files (freeze, adjudication, harness) plus the shared red-test helper module with negative regression tests for each finding. Validated in WSL2 raptor venv (/home/sdrona/raptor/bin/python, Python 3.12.3): targeted suite (freeze_red + adjudication_red + harness_red) 177 passed, 3 skipped; broader tests/eval suite 879 passed, 9 skipped, 2 pre-existing failures in test_tiered_readjudication_frozen_r2.py unrelated to this change (untouched file, fails identically in isolation on this host). No ClinVar/BIAS/Nirvana access performed. PR #26 remains draft. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
… live GET Independent review (round 5) found that despite the round-4 transport- identity-pin defense, scripts/run_clinvar_2026_08_prospective_freeze.py's confirmed --execute path still dynamically imported caller-selected --published-archive-date-lookup/--official-md5-lookup module:callable specs via importlib.import_module, AFTER build_transport() but BEFORE the real GET. Importing any caller-selected module in that window is an arbitrary-code-execution surface in its own right (it can monkeypatch transport internals, issue its own GET, replace execute_transport_and_raw_freeze, or mutate unrelated process state) regardless of the identity pin. - Add src/raptor/eval/prospective_exact_source_metadata_lookups.py: the ONE production-owned, statically-imported implementation of both published_archive_date_lookup/official_md5_lookup ports. Both are permanent fail-closed stubs (MetadataLookupNotYetImplementedError) since neither port's real NCBI source has ever been ratified by this contract; a real implementation must land here via a dedicated, reviewed follow-up, never via a CLI-selected substitute. - scripts/run_clinvar_2026_08_prospective_freeze.py: remove _resolve_import_target and both --published-archive-date-lookup/ --official-md5-lookup CLI options entirely (unrecognized-arguments failure now); replace with static attribute references to the new module. Confirmed live --execute now imports/executes zero caller-selected Python/plugin code before the real GET. The transport- identity pin is retained as defense-in-depth only. - src/raptor/eval/prospective_freeze.py: wrap both lookup-port calls in execute_transport_and_raw_freeze in try/except, converting any raised exception (including the new hard-wired stub's) into a typed BLOCKED/INVALID result (PUBLISHED_ARCHIVE_DATE_LOOKUP_FAILED / OFFICIAL_MD5_LOOKUP_FAILED) instead of a raw propagated traceback; docstrings updated to reflect the closed vulnerability. - tests/eval/test_clinvar_2026_08_prospective_harness_red.py: replace the dynamic-import test fixtures with direct monkeypatching of the new module's attributes; delete tests for the now-nonexistent module:callable resolution/tamper-via-import mechanism; add test_execute_rejects_obsolete_lookup_injection_cli_options (both removed options are unrecognized-argument argparse failures with zero transport/import/network activity) and test_execute_hard_wired_metadata_lookups_reached_with_no_caller_selected_import (real end-to-end --execute reaches the hard-wired stub with import_calls == [] throughout); flip import_calls != [] -> == [] at every remaining call site now that no dynamic import path exists. Validation (WSL2 raptor venv, /home/sdrona/raptor/bin/python 3.12.3): - tests/eval/test_clinvar_2026_08_prospective_harness_red.py: 30 passed - freeze_red + harness_red + adjudication_red combined: 172 passed, 3 skipped - tests/eval full suite: 874 passed, 9 skipped, 2 failed (pre-existing, environment-specific: test_tiered_readjudication_frozen_r2.py ::test_cli_happy_path requires a clean git worktree and ::test_cli_concurrent_publication_fail_closed; unrelated to this change) No ClinVar/BIAS/Nirvana access. PR #26 remains draft. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: 7c146921-f3dd-4a1e-8cf0-8f574de49204
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: 7c146921-f3dd-4a1e-8cf0-8f574de49204
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: 7c146921-f3dd-4a1e-8cf0-8f574de49204
srinivasdrona
marked this pull request as ready for review
September 1, 2026 13:19
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Defines exactly and reproducibly what bytes
x64_freeze.resource_manifest_sha256hashes. Previouslyassert_runtime_boundary(src/raptor/eval/prospective_freeze.py) accepted any 64-character lowercase hex string for this field with no definition of what it must be computed from — the pin was unreproducible.What this adds
raptor.eval.prospective_freeze.compute_resource_manifest_sha256/resource_manifest_entries: a versioned, domain-separated canonical-JSON envelope (schema: raptor.eval.adr0008_resource_manifest_digest.v1) binding, in a fixed pinned order, the identity, order, and raw-byte SHA-256 of the three pinned checksum-manifest files already named inconfigs/eval/core_annotation_bundle.yaml(x64_handoff_requirements.items) and required bydocs/ops/masked-heldout-bias-rerun-handoff.md§4/§9:nirvana-grch38-full.sha256.txtnirvana-grch38-updates.sha256.txtbias-hg38-data.sha256.txtPath.read_bytes()(binary mode, no text-mode newline translation) — the digest is identical on Windows and Linux for byte-identical files, and changes on any filename/order/content drift.docs/ops/adr-0008-resource-manifest-digest.md: the full human-readable algorithm spec, rationale, and operator usage.scripts/compute_adr0008_resource_manifest_sha256.py: a read-only operator CLI for the ADR-0008-designated x64 worker. Reads only the three manifest files, never touches the multi-GB annotation-data bundles, never contacts a network, never runs BIAS/Nirvana, never writes/mutates anything. Refuses to run on a non-x86_64 host unless--allow-non-x64-hostis passed (used only by this repo's own tests).tests/eval/test_adr0008_resource_manifest_digest.py: 16 unit tests covering determinism, content/identity/order sensitivity, fail-closed missing-file handling, a golden-value regression lock on the exact canonical serialization, and CLI integration (including the real refusal path, exercised naturally since this dev host is aarch64).resource_manifest_sha256placeholder comment (still unfilled, still notAPPROVED_PRE_DATA) pointing at the new spec/script.What this intentionally does NOT do
assert_runtime_boundary's behavior — it still only validates the field's format (64-hex-lowercase). No concrete expected value can be pinned in code yet, since it can only be produced by running the new script on the real x64 worker.decisionaway from its non-APPROVED_PRE_DATAplaceholder value.Validation
All commands run in WSL2 Ubuntu,
/home/sdrona/raptor/bin/python, Python 3.12.3,PYTHONPATH=src:A pre-existing, unrelated 5-test failure in
tests/eval/test_tiered_gate_preservation.py/tests/eval/test_tiered_readjudication_frozen_r2.pywas confirmed present before this change too (viagit stash), so it is out of scope here.Do not merge — draft, pending real ADR-0008 x64-worker verification of the checksum-manifest files and a human approval decision.