diff --git a/docs/explain/reference/README.md b/docs/explain/reference/README.md index d58f0793a..78ddbed7c 100644 --- a/docs/explain/reference/README.md +++ b/docs/explain/reference/README.md @@ -1,44 +1,47 @@ -# Reference Notes +# Reference notes -This directory contains implementation-facing reference notes that are not -themselves normative specifications or ADRs. +These notes are for people who work on the code. They are not normative +specs or ADRs. - [coding-standards.md](coding-standards.md) - - Reference coding expectations for the repo + - Coding rules for the repo - [canonical-reference-map.md](canonical-reference-map.md) - - Current canonical locations for SDL, contract, processor, runtime, and - experiment reference material + - Where the main source for SDL, contract, processor, runtime, and + experiment reference material lives now - [documentation-style-guide.md](documentation-style-guide.md) - - Repo-wide documentation style, citation, and tone expectations + - Repo-wide style, citation, and tone rules for docs +- [fixture-provenance.md](fixture-provenance.md) + - How a test fixture that stands in for an external system shows + where its shape came from - [glossary.md](glossary.md) - - Current terminology for SDL, contracts, processing, runtime, experiments, + - Current terms for SDL, contracts, processing, runtime, experiments, and ecosystem concepts - [shared-concept-model.md](shared-concept-model.md) - - Design guidance for `GOV-917` and related shared-concept-model work + - Design guidance for `GOV-917` and related shared concept model work - [shared-semantic-integrity.md](shared-semantic-integrity.md) - - Architecture guardrails for `SEM-200` shared semantic integrity work + - Design rules for `SEM-200` shared semantic integrity work - [backend-conformance.md](backend-conformance.md) - - Architecture guardrails for `ASR-502` backend conformance work + - Design rules for `ASR-502` backend conformance work - [normative-artifact-authority.md](normative-artifact-authority.md) - - Architecture guardrails for `ASR-517` normative authority-boundary work + - Design rules for `ASR-517` normative authority boundary work - [published-schema-coverage.md](published-schema-coverage.md) - - The `ASR-501` rule requiring every published schema to carry corpus, + - The `ASR-501` rule that each published schema must carry corpus, formal, or declared coverage evidence - [assessment-semantics.md](assessment-semantics.md) - - Architecture guardrails for `SEM-206` assessment semantics work + - Design rules for `SEM-206` assessment semantics work - [objective-semantics.md](objective-semantics.md) - - Implementer-facing reference for `SEM-207` declarative-objective semantics + - Reference for those who build `SEM-207` declarative objective semantics - [explicitness-realization-semantics.md](explicitness-realization-semantics.md) - - Architecture guardrails for `SEM-218` explicitness and realization work + - Design rules for `SEM-218` explicitness and realization work - [domain-profiles.md](domain-profiles.md) - - Portable typed profile identity, offline resolution, support negotiation, - opaque preservation, and provenance boundaries + - Portable typed profile names, offline lookup, support checks, opaque + data kept as is, and where data came from - [scenario-variation-and-trial-realization.md](scenario-variation-and-trial-realization.md) - - Complete SCE-002 phase, ownership, identity, random-stream, trial-plan, - runtime-fact, migration, and follow-on architecture + - The full SCE-002 design: phases, owners, names, random streams, trial + plans, runtime facts, migration, and follow-on design - [experiment-binding-contracts.md](experiment-binding-contracts.md) - - Authoritative experiment-to-scenario, participant, and apparatus binding - contracts and configuration realization + - The contracts that bind an experiment to its scenario, participants, + and apparatus, and how its settings are realized - [participant-decision-surface-v2-migration.md](participant-decision-surface-v2-migration.md) - - Reprojection, delivery, capability declaration, admission, reset, memory, - and evidence guidance for participant decision surface v2 + - How to move to version 2 of the participant decision surface: how it + is shown, sent, declared, admitted, reset, kept in memory, and logged diff --git a/docs/explain/reference/fixture-provenance.md b/docs/explain/reference/fixture-provenance.md new file mode 100644 index 000000000..373418e26 --- /dev/null +++ b/docs/explain/reference/fixture-provenance.md @@ -0,0 +1,65 @@ +# Boundary fixture provenance + +A boundary fixture stands in for something this repository does not control. +That can be a platform or third-party API, a command-line tool or subprocess, a +database driver, the filesystem or the operating system, the clock, or another +repository's producer. Such a fixture is evidence only when its shape was +observed from the real system in the state the code handles. A shape written +from documentation or from what the code expects can agree with a wrong +consumer, and the suite then stays green while the behavior cannot work. + +This note sets the rule for adding or changing a boundary fixture. The +[fixture provenance audit](../../research/fixture-provenance/index.md) applied +it to the existing suite (issue #1344). + +## Capture before you encode + +1. Drive the real producer once in the state your code handles and keep what it + returned. One response is enough. Record the producer and its version, the + call or command, the state, the date in UTC, and anything you trimmed. +2. Capture the conditional shapes, not only the happy path: + - an empty or missing value (`null` and `""` are different shapes); + - the error the producer actually raises or prints, with its exit code and + what reaches stdout and stderr; + - counts above one, and repeated runs or retries; + - mixed precision, ordering and time zones in timestamps; + - lifecycle preconditions, such as an operation on an object that is already + stopped or gone. +3. Remove credentials, tokens and signed URL query strings before committing a + capture. + +## Keep the capture where the tests use it + +- Put a capture that tests read in its own file under + `implementations/python/tests/data/`, one file per producer and boundary, + with the provenance fields inside the file. +- Derive the fake's behavior from the capture instead of retyping its shape. + For example, a fake for a library binding loads the capture once, returns the + captured readback and raises the captured error code for a missing object. +- When a test cannot read a capture, state in the fake's docstring where its + shape came from: the capture, the producer's source file, or the published + schema. If no real response could be obtained, say so; that fixture is + inferred and stays listed as such in the inventory. + +## Prove the fake reaches reality + +- Run the production consumer on the captured response in a test. +- Disable the production branch that handles the real shape and rerun the tests. + If they stay green, the fake never exercises that branch. +- Do not build the "real" side of an equality or anti-substitution check through + the code under test. Build it from the producer. + +## Keep captures current + +- Tie each capture to the code that produced its input. A test that re-renders + the input and compares it with the capture flags the capture as stale when + that code changes. +- Re-capture when the producer's version or the input-rendering code changes. +- Never edit a captured artifact by hand. Replace it with a new capture. + +## Review question + +A change that adds or edits a boundary fixture answers one question in the test +module or the capture file: where did this shape come from? "It is what the code +expects" is not an answer. "It is what the producer returned, in this state, on +this date" is. diff --git a/docs/research/fixture-provenance/captures.md b/docs/research/fixture-provenance/captures.md new file mode 100644 index 000000000..f60723218 --- /dev/null +++ b/docs/research/fixture-provenance/captures.md @@ -0,0 +1,323 @@ +# Captured boundary shapes + +These are the responses the #1344 audit observed from real producers. Every +observation was made on 2026-10-09 (UTC) except NS-5, which was read on +2026-10-10. Each entry names the producer and how the response was obtained, +with the producer version where the audit recorded one. Unless an entry says +otherwise, probes ran on macOS 27.0.1 arm64. Long values are trimmed where +marked, and signed URL query strings are removed. Divergence IDs refer to the +register in [`index.md`](index.md). + +The entries for the store and Ground Control client (ST-1 to ST-4), time and +MCP (TM-1 to TM-3) and runtime snapshots (RS-1 to RS-3) are summaries, not raw +responses. Their probes drove this repository's code at `35122105` in its +project environment (CPython 3.14.4), and each entry summarizes what its probe +observed. The probe scripts are not part of this record. + +## GitHub + +GH-1. REST `GET repos/OpenRAE/rae/pulls/25`, read with `gh api` on 2026-10-09, +trimmed to three fields: + +```json +{"base": {"ref": "dev"}, "body": null, "number": 25} +``` + +GH-2. Default branch and a merged PR into `dev`, read with `gh api` and +`gh pr view 1413 --json baseRefName,closingIssuesReferences,body` on 2026-10-09, +trimmed to the fields shown. Issue #1361 was closed by `github-actions[bot]`, +not by the merge. `closes_lines` is not a `gh` field: it lists the +`Closes #N` text in `body`, as the jq expression +`[.body | scan("Closes #[0-9]+")]` extracts it: + +```json +{"default_branch": "main"} +{"baseRefName": "dev", "closes_lines": ["Closes #1361"], "closingIssuesReferences": []} +``` + +GH-3. Release run 36967479436, job `publish-github`, log lines 361-363 (tag `v6.0.1`): + +```text +failed to run git: fatal: not a git repository (or any of the parent directories): .git +##[error]Process completed with exit code 1. +``` + +GH-4. `gh attestation verify` 2.101.0 output for the v6.0.1 wheel, trimmed to the +identity the verifier reads. The list held one entry. The same command on a +bundle that held this attestation twice exited 0 and printed two entries with +this identity: + +```json +[{"verificationResult": {"signature": {"certificate": { + "issuer": "https://token.actions.githubusercontent.com", + "sourceRepositoryURI": "https://github.com/OpenRAE/rae", + "buildSignerURI": "https://github.com/OpenRAE/rae/.github/workflows/release-please.yml@refs/heads/main", + "runInvocationURI": "https://github.com/OpenRAE/rae/actions/runs/36967479436/attempts/1"}}}}] +``` + +GH-5. A partially re-run workflow run, `GET actions/runs/36342459179` and its +attempt-2 job list, read on 2026-10-09. Jobs carried over from attempt 1 keep +their attempt-1 start time: + +```json +{"run_attempt": 2, "created_at": "2026-09-27T18:55:46Z", "run_started_at": "2026-09-27T19:05:09Z"} +[{"name": "canonical / generic-tool-local-inputs", "started_at": "2026-09-27T19:05:14Z"}, + {"name": "canonical / test-shard (0)", "started_at": "2026-09-27T18:55:49Z"}, + {"name": "fast-feedback", "started_at": "2026-09-27T18:55:54Z"}] +``` + +NS-3. The v6.0.1 release evidence produced by run 36967479436, from its +`release-evidence-3d59d0eb64d3736e337a387f55bc6f1880ccfcfd` artifact (ID +11210897140), downloaded with `gh run download`. The workflow keeps that +artifact for 7 days. It expired at 05:29Z on 2026-10-09, after the download, so +this capture cannot be fetched again. `release-evidence-index.json`, trimmed: + +```json +{"release": {"run_attempt": "1", "run_id": "36967479436", "tag": "v6.0.1", + "source_sha": "3d59d0eb64d3736e337a387f55bc6f1880ccfcfd", + "workflow_sha": "3d59d0eb64d3736e337a387f55bc6f1880ccfcfd"}, + "subjects": [{"role": "wheel", "sha256": "3e97f42d42564acc1740757157ffd35a8c6373487845e987a314da54388a06c0"}, + {"role": "sdist", "sha256": "8ab156fb1b9c1f0b467c08d98c0695403224ad3c6f2b7835e30b570f8741e67f"}], + "test_subjects": [{"role": "derived-test-wheel", "derived_from": "raes-6.0.1.tar.gz", + "sha256": "3e97f42d42564acc1740757157ffd35a8c6373487845e987a314da54388a06c0"}]} +``` + +The artifact's `build-inventory.json` lists all three subjects (`wheel`, `sdist`, +`derived-test-wheel`), and the derived test wheel has the wheel's digest. The +index's `release` block is also the observation behind the inventory's +`GITHUB_SHA` and `GITHUB_WORKFLOW_SHA` row. + +## libvirt + +LV-1 and LV-4. libvirt 12.7.0 `test:///default` through libvirt-python 12.8.0, +captured on 2026-10-09: + +```text +defineXML, same name, new UUID: libvirtError code=9 "operation failed: domain 'raes-audit-web' already exists with uuid 19837e5e-7699-51af-8bd1-435e6ce85a5f" +defineXML, same UUID, new name: libvirtError code=9 "operation failed: domain 'raes-audit-web' is already defined with uuid 19837e5e-7699-51af-8bd1-435e6ce85a5f" +destroy() on an inactive domain: libvirtError code=55 "Requested operation is not valid: domain is not running" +``` + +LV-2 and LV-3. The same libvirt and binding, captured at 09:48Z. The domain and +network XML was rendered at `35122105` by `native_matrix`, `network_xml` and +`domain_xml` in `raes_backend_libvirt.techvault_matrix`. Each object was +defined and started, then read back with `XMLDesc(0)`. The domain's changed +elements, trimmed: + +```text +defined: 128 2 +read back: 131072 + 131072 + 2 +``` + +Each network read back with an added ``. Lookups of +names that were never defined, with the name elided: + +```text +lookupByName: libvirtError code=42 "Domain not found" +networkLookupByName: libvirtError code=43 "Network not found: no network with matching name ''" +``` + +`test:///default` starts no hypervisor, so a `qemu:///system` readback adds +elements this capture does not show, such as device addresses and aliases. +#1453 proposes the full capture, with the rendered XML, as test data for the +TechVault fakes. + +## Guest appliance + +GU-1. Serial fact channel of the production-built guest appliance, booted with +128 MiB under `qemu-system-x86_64` (Ubuntu noble `busybox-static` 1.36.1, kernel +`linux-image-6.8.0-1067-aws`). The bytes end in CRLF (`^M`): + +```text +RAES-GUEST-FACTS v1^M +challenge 3904beced86cd1db84d3f7137b7a432c^M +architecture x86_64^M +vcpus 1^M +memory_mib 81^M +iface 52:54:00:c4:7e:78 192.0.2.10 1^M +content /etc/raes/marker aabb38db6cd478c6dcbf41b2973d99a8e72cafe9a026df5808d5015452b38a55 644^M +service beacon tcp 9000 1 1^M +init complete^M +``` + +The kernel log of the same boot: + +```text +Memory: 74496K/130552K available (22528K kernel code, 4446K rwdata, 14088K rodata, 5048K init, 4656K bss, 55796K reserved, 0K cma-reserved) +``` + +GU-2. Account lines from a 128 MiB boot that requested `loner`, `ops` and `dflt` +enabled and `analyst` disabled: + +```text +account loner 1000 /home/loner /bin/sh 1 ^M +account analyst 1001 /home/analyst /bin/sh 1 raes^M +account ops 1002 /home/ops /bin/sh 1 raes,wheel^M +account dflt 1003 /home/dflt /bin/sh 1 ^M +``` + +## Network services + +NS-1. macOS `/usr/bin/curl` 8.7.1 with the production argv shape +(`--fail --location --proto =https --max-filesize`) on the locked ATLAS asset +(625790 bytes) with a limit one byte short, captured on 2026-10-09: + +```text +--http2 exit=56 curl: (56) Maximum file size exceeded +--http1.1 exit=63 curl: (63) Maximum file size exceeded +exact limit (625790) exit=0, 625790 bytes +``` + +Ubuntu 24.04 curl 8.5.0 returned 63 for the same transfer on both protocols. + +NS-2. Anonymous tag listing on public registries, captured on 2026-10-09: + +```text +GET https://ghcr.io/v2/oras-project/oras/tags/list +HTTP/2 401 +www-authenticate: Bearer realm="https://ghcr.io/token",service="ghcr.io",scope="repository:oras-project/oras:pull" + +GET https://registry-1.docker.io/v2/library/alpine/tags/list +HTTP/2 401 +www-authenticate: Bearer realm="https://auth.docker.io/token",service="registry.docker.io",scope="repository:library/alpine:pull" +``` + +With a token, a ghcr.io blob request returns `HTTP/2 307` with +`location: https://pkg-containers.githubusercontent.com/ghcr1/blobs/sha256:...?`. + +NS-4. Two downloads of the locked NIST CSF export URL returned 107517 and 107518 +bytes with different SHA-256 digests. The workbook's `docProps/core.xml` +creation time and one sheet cell differ between them, and neither digest matches +the lock. + +NS-5. The Docker Official Image `tomcat:latest` (Tomcat 11.0.26), read +anonymously on 2026-10-10 from `public.ecr.aws/docker/library/tomcat`. `HEAD` +requests to `registry-1.docker.io` returned the same index digest +(`sha256:bb7c0c78078841047e2d475acce833ef337cb0007ca7c7e4c9ff9058d9019801`) and +the same linux/amd64 manifest digest +(`sha256:594377d00cebafa7a413a171ad35bd1756c9be9b26855c8c3ba23313db4e18b9`). +That manifest lists ten layers, and three of them are one 32-byte layer: + +```text +layers[4] sha256:4f4fb700ef54461cfa02571ae0db9a0dc1e0cdb5577484a6d75e68dc38e8acc1 size=32 +layers[7] sha256:4f4fb700ef54461cfa02571ae0db9a0dc1e0cdb5577484a6d75e68dc38e8acc1 size=32 +layers[9] sha256:4f4fb700ef54461cfa02571ae0db9a0dc1e0cdb5577484a6d75e68dc38e8acc1 size=32 +``` + +## Command-line tools + +CL-1. The pinned conftest 0.68.0 (OPA 1.15.1), `conftest test input.json --policy --output json`, +captured on 2026-10-09: + +```text +policy that passes: exit=0, stdout [{"filename": "input.json", "namespace": "main", "successes": 1}] +no rules in main: exit=0, stdout [{"filename": "input.json", "namespace": "main", "successes": 0}] +rule conflict: exit=1, stdout empty, stderr + Error: running test: query rule: check: query rule: evaluating policy: policy-conflict/conflict.rego:6: eval_conflict_error: complete rules must not produce multiple outputs +parse error: exit=1, stdout empty, stderr + Error: running test: load: loading policies: load: 1 error occurred during loading: policy-parse-error/broken.rego:9: rego_parse_error: unexpected eof token +conftest verify: exit=1 on the parse error, exit=0 on the rule conflict +``` + +CL-2. git 2.54.0 (Apple Git-157), `git diff --name-only --diff-filter=d --cached` +in a scratch repository, next to the `-z` form: + +```text +"contracts/schemas/na\303\257ve.json" +contracts/schemas/plain.json +docs/renamed.md +"docs/tab\tname.md" +docs/with space.md + +git ls-files -z: contracts/schemas/naïve.json | contracts/schemas/plain.json | docs/renamed.md | docs/tabname.md | docs/with space.md +``` + +## Control-plane store and requirements client + +ST-1. `flock`, observed through a forked child and a separate probe process. The +first two lines come from an OS-level `flock` probe. The last two run the +production `RuntimeOwnerLease.close()` and a variant that always unlocks: + +```text +after forked child closed a dup: BlockingIOError(errno=35 EAGAIN) +after forked child called LOCK_UN on inherited fd: acquired (parent lock was silently released) +production close() in a forked child, third process: blocked (parent still exclusive) +close() that always unlocks, third process: ACQUIRED the lock (two owners possible) +``` + +ST-2. SQLite 3.50.4 under CPython 3.14.4, a commit-time failure in the +production `transaction()`: + +```text +deferred foreign key: transaction() raised from commit(): sqlite3.IntegrityError(FOREIGN KEY constraint failed) + in_transaction after failed commit: True; row persisted after close(): 0 +WAL write failure at COMMIT (RLIMIT_FSIZE): sqlite3.OperationalError(disk I/O error) SQLITE_IOERR_WRITE + in_transaction after failure: False; PRAGMA quick_check: ok +disk full inside the body: sqlite3.OperationalError(database or disk is full) SQLITE_FULL, isinstance(OSError) False +``` + +ST-3. A production store file under SQLite 3.50.4, corrupted in place, then +admitted: + +```text +index root page overwritten: quick_check row "Tree 10 page 10: btreeInitPage() returns error code 11" + admit_runtime raises ValueError(local control-plane database failed its integrity check) +state table root page overwritten: quick_check raises sqlite3.DatabaseError(database disk image is malformed) + admit_runtime raises sqlite3.DatabaseError(database disk image is malformed) +file truncated from 40960 to 20480 bytes: same as above +``` + +ST-4. The production Ground Control client against local test servers: + +```text +connection refused: GroundControlUnavailable (handled) +server closes without response: http.client.RemoteDisconnected (not a GroundControlError) +truncated body: http.client.IncompleteRead (not a GroundControlError) +200 with an HTML page: json.decoder.JSONDecodeError (not a GroundControlError) +200 with non-UTF-8 JSON: UnicodeDecodeError (not a GroundControlError) +401 JSON: GroundControlAuthRequired (handled) +``` + +## MCP, time and runtime payloads + +TM-1. Workflow steps completed 1 ms apart, timestamps formatted the way the RAES +runtime (`datetime.now(UTC).isoformat()`) and the LilRAE workflow engine format +them: + +```text +A = 2026-10-08T12:00:01Z B = 2026-10-08T12:00:01.001000Z +compensation order recorded: ['a', 'b'] (reverse completion order is ['b', 'a']) +control with both fractions present: ['b', 'a'] +``` + +TM-2. `RuntimeFactBindingAdmission` and `bind_action_inputs`: + +```text +'2026-07-20T03:02:00Z': admitted; bind accepted, disposition BOUND +'2026-07-20T03:02:00': admitted; bind raised TypeError: can't compare offset-naive and offset-aware datetimes +'2026-07-20': admitted; bind raised TypeError: can't compare offset-naive and offset-aware datetimes +``` + +TM-3. The `raes_mcp` server over stdio, `sdl_validate` called with an object +where the schema expects a string: + +```text +isError: True +Error executing tool sdl_validate: 1 validation error for sdl_validateArguments +sdl_content + Input should be a valid string [type=string_type, input_value={'secret': 'do-not-disclose-42'}, input_type=dict] +``` + +RS-1, RS-2 and RS-3. One live control plane (stub target), compared across its +producers: + +```text +carriers in the /snapshot API but absent from the conformance hermetic payload: realization_envelope, realization_provenance +store serializer closure records: present; /snapshot route closure records: {} +mixed_composition_states and mixed_composition_history: present in the store, {} in the public API +real snapshot with participant behavior history, semantic conformance: + participant behavior history requires a participant.behavior snapshot entry with action_contract_addresses and observation_boundary_addresses + participant behavior event references unknown action_contract_address 'participant.action-contract.scan' +``` diff --git a/docs/research/fixture-provenance/index.md b/docs/research/fixture-provenance/index.md new file mode 100644 index 000000000..0b5dd389f --- /dev/null +++ b/docs/research/fixture-provenance/index.md @@ -0,0 +1,388 @@ +# Fixture provenance audit + +Issue #1344 asks which test fixtures assert shapes the real system never +produces. This record answers it for this repository on `dev` at `35122105` +(2026-10-09). For each boundary the audit examined where a test stands in for +something the repository does not control, it records whether the fixture's +shape was captured or inferred. It checks the shapes that matter against the +real producer and lists the divergences found, with their status. It is +research evidence, not contract authority. + +- [`inventory.md`](inventory.md) lists each boundary with its provenance, the + real observation, a verdict and a blast-radius rating. +- [`captures.md`](captures.md) holds the observed responses. +- [Boundary fixture provenance](../../explain/reference/fixture-provenance.md) + is the convention for new boundary fixtures. + +Issue #1344 is labelled `cross-repo`. This audit covers only this repository. +External producers from sibling repositories (LilRAE's APTL backend) appear only +where this repository consumes their output. + +## Method + +1. Inventory. The fakes, stubs, monkeypatched callables, hand-built payloads and + corpus files that the audit found under `implementations/python/tests/` and + `contracts/fixtures/` were grouped into ten boundary classes: GitHub + platform, libvirt API, guest appliance, network services and supply chain, + subprocess tools, the SQLite store and Ground Control client, MCP and + filesystem and time, backend manifests, runtime snapshots, and release and + experiment evidence. +2. Provenance. Each fixture was traced to the production consumer that relies + on it and labelled `captured`, `inferred`, `synthetic` or `real`. No GitHub + fixture records where its shape came from, and no libvirt or guest fixture + points to a captured response. +3. Observation. The audit drove the real producer in the state the consumer + handles and recorded what came back: + - read-only GitHub REST and `gh` 2.101.0 calls against `OpenRAE/rae`; + - libvirt 12.7.0 `test:///default` with production-rendered XML; + - boots of the production-built guest appliance under `qemu-system-x86_64`; + - the pinned tool binaries (conftest 0.68.0, gitleaks 8.30.1, osv-scanner + 2.4.0, vale) and Docker 29.7.2; + - SQLite 3.50.4 under CPython 3.14.4; + - local servers that fail the way networks fail; + - LilRAE at `a5833df9`, read only, for its manifest, snapshot and evidence + shapes. +4. Check the check. For the libvirt fakes and the store lease, production + already handled the real shape, so the audit disabled the branch that handles + it and reran the suite. A suite that stays green under that change has fakes + that never reach reality. + +Limits: + +- Most probes ran on macOS arm64. curl was also checked in an Ubuntu 24.04 + container. +- No live `qemu:///system` daemon was available. +- GitHub was read, never written. Transferred-issue redirects and `open` release + assets could not be produced. +- Fixture counts were not recorded for the subprocess, SQLite store, MCP and + runtime-snapshot classes; the inventory lists their boundaries instead. +- Probes for store migration across releases and request-commitment drift + recorded no verdict. +- The inventory is not exhaustive. Stand-ins the audit did not examine + include: + - the pinned z3 binding: `z3.Solver.check` and `reason_unknown` are patched + at `T/test_scenario_satisfiability.py:353-365`, `:414` and `:692-700`, and + `P/raes_processor/satisfiability/_solver.py:254-257` consumes both; + - the solver's `time.monotonic_ns` fakes at + `T/test_scenario_satisfiability.py:526`, `:659`, `:678`, `:699` and `:724`, + which `_solver.py:18-19` and `:59-73` read; + - the TechVault host probe stub at + `T/test_libvirt_backend_techvault_native.py:196-201`, which + `check_native_readiness` discards; + - podman, the OCI driver's second allowed runtime + (`P/raes_reference_backend/drivers/oci.py:43`), whose output was not + observed. + +## Results + +| Class | Fixtures examined | Diverges | Unverified | +|---|---|---|---| +| GitHub platform and workflow runtime | 67 | 6 | 3 | +| libvirt Python API | 98 fake definitions (16 surfaces) | 4 | 2 | +| Guest appliance | 31 | 4 | 2 | +| Network services and supply chain | 171, plus 82 lock values | 7 | 2 | +| Subprocesses and command-line tools | not recorded (12 boundaries) | 2 | 1 | +| SQLite store and Ground Control client | not recorded (7 boundaries) | 4 | 1 | +| MCP, filesystem, time and internal stand-ins | not recorded (7 boundaries) | 3 | 0 | +| Backend manifests and realization envelopes | 52 | 3 | 1 | +| Runtime snapshots and control-plane payloads | not recorded (5 boundaries) | 3 | 0 | +| Release, conformance and experiment evidence | 29 | 1, plus NS-3 | 0 | + +Most boundaries match. The divergences cluster where a field or behavior is +conditional: + +- an empty value that arrives as `null`; +- a count that can exceed one; +- a re-run that mixes attempt numbers; +- an error that exits 1 with nothing on stdout; +- a timestamp that loses its fraction on a whole second; +- a lifecycle precondition that a dictionary fake cannot express. + +## Divergence register + +Each entry gives the fixture where there is one, the shape it asserts and what +the real producer returned. It also names the production code that relies on +that shape, the recommended fix and the status. "Open" means no change in this +repository fixes it yet, and each open entry names the issue that tracks it. +The pull requests cited below were open on 2026-10-12, except #1415, which +merged on 2026-10-11. + +### Addressed by pull requests + +- **LV-2: libvirt missing objects raise `KeyError` in the TechVault fakes.** + Blast M. Fixtures: `T/test_libvirt_backend_techvault_native.py:105-109` and the + same fakes in `T/test_libvirt_backend_guest_certified.py` and + `T/test_libvirt_evidence_run.py`. Real: `libvirt.libvirtError` code 42 or 43. + Production handles code 42/43 (`_techvault_native_ops.py:57`, + `techvault_lifecycle.py:84`), but no hermetic test reached those branches: + disabling either left all 128 tests in the three modules passing. Fix: the + fakes raise the captured codes. Status: #1453 (test only). +- **LV-3: `XMLDesc` echoes the defined XML.** Blast M. Same fixtures. Real: + memory reads back as `131072` with + `currentMemory`. Production converts KiB correctly + (`techvault_observation.py:196`), but no test fed it KiB. Fix: the fakes derive + their readback from the capture, and a test runs the production readers and + concern gate on the captured XML. Status: #1453 (test only). +- **CL-1: conftest error output read as "no failures".** Blast H (policy gate). + Fixture: `T/test_repo_policy_tools.py:1288-1316` models only failure output. + Real: conftest 0.68.0 exits 1 with an empty stdout when it cannot load or + evaluate the policy. Production: `tools/policy/conftest_tool.py:86-91` returns + `[]` for an empty stdout. Fix: raise the existing evaluation error. Status: + #1455, issue #1454. +- **ST-1: the runtime-owner lease fork test cannot see the non-owner unlock + guards.** Blast H (single-owner store). Fixture: + `T/test_issue_1092_control_plane_crash_consistency.py:2432` + (`test_runtime_owner_lease_rejects_and_closes_in_a_different_process_identity`) + patches `os.getpid` in one process. Real: after `fork()` a child that unlocks + its inherited descriptors releases the parent's `flock`. Production guards are + correct (`control_plane_store_lease.py:217-231`, `:113-120`), but removing + either still passes the test. Fix: a test that holds duplicates of the lease + descriptors and probes both locks. Status: #1459 (test only), issue + #1558. +- **TM-1: compensation order sorts timestamps as strings.** Blast H. Fixtures: + the compensation tests in `T/test_runtime_control_plane_api.py` have one + compensable step, so none observes the order. Real: RAES + (`control_plane_execution.py:39-40`) and the LilRAE workflow engine format + instants with `isoformat()`, which drops the fraction on a whole second, so + `...:01.001000Z` sorts before `...:01Z`. Production: + `control_plane_workflows.py:52` sorts the text, against the reverse-completion + rule in `specs/formal/workflows/compensation.md`. Fix: sort parsed instants. + Status: #1464, issue #1462. +- **GH-3: `gh release upload` in a job without a checkout.** Blast H. Issue + #1414. Status: fixed by #1415, merged on 2026-10-11. +- **GH-7: `run_started_at` resets on re-run.** Blast L. Fixture: + `T/test_ci_latency_report.py:12-123`. Production: + `tools/ci_latency_report.py:129-137` reports the re-run delay as queue time. + Fix: record the attempt and measure attempt 1 separately. + Status: #1491, issue #1490. +- **NS-1: curl over HTTP/2 on macOS.** Blast M, macOS profile only. Fixture: + `_CurlFixture` in `T/test_issue_1217_bootstrap_profiles.py:776-819` speaks + HTTP/1.x only. Real: macOS curl 8.7.1 exits 56 for a size-limit breach and does + not retry a 503 over HTTP/2. Production: + `tools/maintained_client_acquisition.py:236-243` recognizes only 63, and + `:146-151` relies on curl's retries. Fix: pass `--http1.1`, the protocol the + qualification covers. Status: #1476, issue #1475. +- **NS-2: OCI registries require a Bearer token.** Blast M. Fixture: + `T/test_sdl_module_registry.py:96-170` serves every endpoint anonymously. + Real: ghcr.io, Docker Hub and ECR Public answer 401 with a `Bearer` challenge, + and blobs redirect with 307. Production: `raes/module_registry/__init__.py:156-186` + treats the 401 as a fetch error. Fix: implement the anonymous token flow + without forwarding the token to the redirect target. + Status: #1480, issue #1479. +- **MF-2: the mixed-runtime check compares internal representations.** Blast M. + Fixture: `T/test_issue_1016_mixed_runtime_coordination.py:278-291` builds the + target through the same round trip the check uses. Real: the APTL manifest's + authored OS order and list type differ from the round trip. Production: + `P/raes_runtime/mixed_runtime.py:212-214`. Fix: compare canonical contract + forms. Status: #1483, issue #1482. +- **CL-2: git path listings without `-z`.** Blast M, latent (no non-ASCII path + is tracked today). Real: git quotes such paths. Production: `_changed_paths` + and `_tracked_repo_paths` in `tools/nox_support/runner.py` drop them, and + `changed_paths` in `tools/policy/common.py` returns the quoted text. Fix: use + `-z` listings. Status: #1474, issue #1473. +- **ST-2: commit-failure fixtures raise inside the transaction body.** Blast L. + Fixtures: `T/test_run_310_supervisory_lifecycle.py:872` and + `T/test_runtime_control_plane_api.py:1878` raise `OSError` inside the + transaction body. Real: commit failures raise `sqlite3` errors from + `commit()`. Atomicity holds on real SQLite because the connection is closed + after each operation. Status: #1472 (test only), issue #1559. +- **NS-7, NS-8: inert or harmless fixture drift.** Blast L. Fixtures: + `T/test_issue_1226_attestation_verifier.py:74-77` and the curl `000` stub in + `T/test_release_workflows.py:1394-1411`. Status: #1488 (test only), issues + #1560 (NS-7) and #1561 (NS-8). + +### Open: production change needed + +- **GH-1: an empty PR description is `null`.** Blast L. Fixtures: + `T/test_pr_body_guard.py:633`, `T/test_pr_body_policy_migration.py:61` use + `""`. Production: `tools/check_pr_body.py:467-469` raises a configuration error + (exit 2) for `null` before the automation exemption runs. Fix: treat `null` as + an empty body and evaluate the exemption first. The validator file is + byte-pinned by the #1284 policy migration (`pr-body-policy.yml` and + `T/test_pr_body_policy_migration.py`), so the fix also moves that pin. + Status: open, issue #1525. +- **GH-2: GitHub ignores closing keywords in PRs that target `dev`.** Blast M. + Fixture comment: `T/test_pr_body_guard.py:295-296`. Real: PR #1413 says + `Closes #1361` and has `closingIssuesReferences: []`. Production: + `tools/check_pr_body.py:164-187` describes its pattern as "the closing + references GitHub reads". Keywords in squashed commit messages do act once + `dev` reaches `main`, and nothing checks them. Fix: check commit messages for + closing references in `tools/check_pr_title.py`, which already receives them, + and correct the docstring. Status: open, issue #1527. +- **GH-4: more than one verified attestation per artifact.** Blast H. Fixture: + `T/test_issue_1226_attestation_verifier.py:99-103`. Real: gh 2.101.0 prints one + entry per verified attestation, and re-attesting byte-reproducible builds adds + entries with the same identity. Production: + `tools/release_evidence_verifier.py:107-111` refuses any list longer than one + (`verifier-ambiguous-attestation`), so a re-run or repair of an attested tag + can never be admitted. Fix: refuse only distinct identities. + Status: open, issue #1529. +- **GH-5: a partial re-run mixes attempt numbers.** Blast M. Fixture: + `T/test_issue_1226_release_admission.py:283-287` treats another attempt as a + replay. Real: run 36342459179. Production: + `tools/release_evidence_admission.py:398-408` requires exact attempt equality, + so "re-run failed jobs" after a transient admission failure cannot pass. Fix: + keep `run_id` strict and accept `1 <= index attempt <= GITHUB_RUN_ATTEMPT`. + Status: open, issue #1531. +- **LV-1: the generic driver identifies native objects by name only.** Blast H. + Fixture: `T/test_libvirt_backend_driver.py:111-125` stores objects in a + name-keyed dictionary that accepts any UUID. Real: a name or UUID collision + raises code 9, so a RAES-owned object can live under another name (after a + `name_prefix` change, a provider-name change or a rename). Production: + `deployment.py:428-430` and `:464-490` treat a name miss as absence; destroy + then reports the object removed while it keeps running. Fix: fall back to + `lookupByUUIDString` and `networkLookupByUUIDString` on the owned UUID, and + make the fakes raise code 9 on collisions. Status: open, issue #1533. +- **GU-1: the guest reports far less memory than the domain has.** Blast H. + Fixtures: `T/test_libvirt_backend_guest_certified.py:115` (120), `:490`, `:683`. + Real: `memory_mib 81` from `MemTotal` at 128 MiB; the committed July evidence + says 78. Production: `guest_observation.py:65` and `:245-246` accept 112-128, + so a real guest fails certification. Fix: measure firmware "System RAM" in the + probe and bump the probe policy, or derive the tolerance from measurements. + Status: open, issue #1535. +- **GU-2: the guest's account `disabled` report is constant.** Blast H. + Fixtures: `T/test_libvirt_backend_guest_certified.py:183`, `:600`, `:608`. + Real: every account reports `1`. Production: enabled accounts are written with + `*` (`guest_appliance.py:178`) and the report treats `*` as disabled (`:246`); + the SDL pipeline also sets `lock_passwd=True` for every account + (`realization/_cloud_init.py:122`) and `techvault_matrix.py:132` maps that to + `disabled`. Fix: report only `!` as disabled and carry the authored `disabled` + separately from `lock_passwd`. Status: open, issue #1537. +- **GU-3: cloud-init reads `user-data` as YAML.** Blast M. Fixture: + `T/test_libvirt_backend_cloudinit.py:28-31` checks with `json.loads`. Real: + PyYAML turns `json.dumps` surrogate pairs into lone surrogates, and + cloud-init's UTF-8 write fails. Production: `cloudinit.py:120`. Fix: emit YAML + `\U` escapes for astral-plane characters and use `yaml.safe_load` as the oracle. + Status: open, issue #1539. +- **TM-2: naive `requested_at` values pass admission.** Blast M, latent: only + tests construct the admission today. Fixtures: + `T/test_runtime_fact_bindings.py:107` and `:233-243` use `Z` timestamps. + Real: a value built with `datetime.isoformat()` from a naive datetime, or a + date-only value, is admitted (`runtime_fact_dispatch.py:36`) and binding + raises `TypeError` (`runtime_fact_binding_policy.py:119-124`). Fix: require an + offset at admission. Status: open, issue #1541. +- **NS-3: the sdist-built test wheel equals the wheel.** Blast M. Fixtures: + `T/test_issue_1226_release_admission.py:70`, `:172` and + `T/test_issue_1227_release_publication.py:64`, `:156` use distinct bytes and a + 2-subject inventory. Real: 3 subjects, and the derived wheel has the wheel's + digest. Production: `tools/release_evidence_admission.py:314-320` passes only + because the digests coincide. Fix: admit the derived subject against + `test_subjects`, or make byte identity an explicit gate. + Status: open, issue #1545. +- **EV-1: LilRAE evidence records omit `capture_spec_ref.ref_version`.** Blast M. + Fixtures: the 5 files under + `contracts/fixtures/experiment-core/experiment-evidence-record-v1/valid/`. + Real: the schema accepts LilRAE's record and + `raes_contracts/evidence_satisfaction.py:98-107` rejects it. Fix: compare as + `run_ref` already does, or require the version in the schema and in LilRAE. + Status: open, issue #1547. +- **MF-1: processor and backend compatibility exists only in fixtures.** Blast M. + Fixtures patch the pairing: `compatibility.backends` at + `T/test_sce_002_trial_realization.py:67` and `T/test_runtime_contracts.py:81`, + and the backend name at `T/test_runtime_contracts.py:91`. Real: the reference processor + declares `["stub"]`, so every shipped non-stub backend is rejected by + `trial_realization.py:224-228` and `trial_compiler/apparatus.py:389-401`. Fix: + declare the certified pairings or document that trial realization is stub-only. + Status: open, issue #1549. +- **RS-1, RS-2, RS-3: conformance and the SEM-222 test validate projections that + differ from the published snapshot.** Blast M. Real, from one live control + plane: the hermetic conformance payload omits `realization_envelope` and + `realization_provenance`, which the `/snapshot` API publishes (RS-1); the + public route drops the closure and mixed-composition state that the store + serializer keeps, and the SEM-222 test reads the store serializer (RS-2); a + `/snapshot` with participant behavior history, set up as in + `T/test_runtime_control_plane.py`, fails semantic conformance because it has no + `participant.behavior` entry, which the corpus supplies with an empty payload + (RS-3). Fix: validate the payload the public producers emit, and build corpus + snapshots from the real pipeline. + Status: open, issues #1551 (RS-1), #1553 (RS-2) and #1555 (RS-3). +- **ST-4: Ground Control transport failures escape classification.** Blast M. + Fixture: `T/test_requirement_governance.py:150-152` fakes `HTTPError`, + `URLError` and `TimeoutError`. + Real: dropped connections, truncated bodies, HTML login pages and non-UTF-8 + bodies raise other exceptions. Production: `tools/policy/requirement_governance.py` + maps only some failures to `GroundControlError`. Fix: classify transport and + decoding errors as unavailable. Status: open, issue #1557. +- **TM-3: rejected MCP arguments echo the input value.** Blast L. + `T/test_experiment_authoring.py:195-206` asserts no `input_value` in the + rendered result, passing `spec_content` as a string. Real: an object argument + gets the MCP framework's validation error with `input_value`. Fix: handle + argument validation errors in the server. Status: open, issue #1543. + +### Open: smaller corrections + +Blast L unless marked. Status: open, tracked together in #1521. + +- **LV-4:** generic-driver fakes in `T/test_libvirt_backend_driver.py`: + `destroy()` succeeds on an inactive object (real: code 55) and `undefine()` + leaves the object findable. The fakes, and the comment at + `deployment.py:392-393`, also treat `nwfilterDefineXML` as an unconditional + upsert. libvirt's `virNWFilterObjListAssignDef` (`src/conf/virnwfilterobj.c`) + rejects a name or UUID mismatch with code 9, but the test driver has no + nwfilter driver, so that part is unverified. +- **GU-4 (blast M):** the committed guest evidence artifact fails the current validator + with 2 problems and no test loads it. Fix: recapture it or mark it historical. +- **NS-4:** the NIST CSF export is regenerated per request, so `--verify-remote` + always fails. Fix: lock the canonical digest the checker already computes. +- **NS-5:** `tools/tooling_artifact_policy_oci.py:102-108` rejects repeated layer + digests, which real manifests contain. +- **NS-6:** inert fixture drift in `T/test_oci_release_image.py:122-124`. + Nothing reads its fields, so the inventory rates it `n/a` and the results + table does not count it as a divergence. +- **MF-3:** 9 of the 17 negative corpus files under + `contracts/fixtures/backend-manifest/backend-manifest-v2/invalid/` (checked by + `T/test_backend_manifest.py:1179-1183`) fail for incidental reasons as well as + their named defect. Fix: derive each from the current valid stub with one + mutation and assert the reason. +- **ST-3:** the `_QuickCheckFailureConnection` fake + (`T/test_issue_1092_control_plane_crash_consistency.py:1028`) returns a + non-`ok` row, but severe corruption raises `sqlite3.DatabaseError` from + `PRAGMA quick_check` instead of the store's integrity error; admission still + refuses. + +## What remains + +- The open divergences above need production changes. GH-2, GH-4, GH-5, NS-3, + MF-1, GU-1 and GU-2 also involve a release-admission or semantics choice that + belongs to the maintainer, and their issues (#1527, #1529, #1531, #1545, + #1549, #1535 and #1537) list the options. +- Unverified boundaries: transferred-issue redirects, `GITHUB_SHA` on + `workflow_dispatch`, `open` release assets, libvirt code 62 for a missing + nwfilter under current code, libvirt code 9 for an `nwfilterDefineXML` name or + UUID collision, fact-file ownership under a live libvirtd, + `genisoimage`, the AWS EC2 CLI JSON, lock digests for self-downloading actions, + `check-jsonschema`, store migration across releases, and capture offers. +- Boundaries not examined, such as the z3 solver binding (see Limits). +- #1522 tracks the unverified boundaries and the boundaries not examined. +- Findings for sibling repositories are listed below. Filing them in those + repositories is outside this change. + +## Findings for sibling repositories + +The failure class is not specific to this codebase. These observations apply +wherever LilRAE, env-packs, adapters or hub use the same platform, tool or +pattern: + +- Every sibling's default branch is `main`, so closing keywords in PR bodies that + target `dev` close nothing, while keywords in commit messages act on promotion. +- Workflow-run `pull_requests` lists open PRs only; a Phase E workflow triggered + on `pull_request: closed` sees `[]`. LilRAE has the same Phase E workflow. +- `pull_request.body` can be `null`, and `run_started_at` resets on re-run. +- A `gh` call without `--repo` in a job without a checkout fails with + `not a git repository`. `gh api` paths with `{owner}` or `{repo}` resolve + through git too. +- Release attestations accumulate per digest; a verifier must not treat several + attestations with one identity as ambiguous, nor bind to one run attempt across + jobs (env-packs attests releases). +- libvirt reports missing objects as `libvirtError` codes 42 and 43 (observed) + and, per libvirt source, 62 for a missing nwfilter. It treats name and UUID + as separate identities and returns normalized XML. A fake that + classifies errors by type must register its error class as + `sys.modules["libvirt"].libvirtError`. +- `http.server` fixtures speak HTTP/1.x only; macOS curl changes exit codes and + retries over HTTP/2. +- Public OCI registries need the anonymous Bearer token flow. +- conftest exits 1 with an empty stdout on load and evaluation errors. +- `datetime.isoformat()` drops the fraction on a whole second, so timestamps from + one producer must be compared as instants, not strings. +- Never build the "real" side of an equality check through the code under test. diff --git a/docs/research/fixture-provenance/inventory.md b/docs/research/fixture-provenance/inventory.md new file mode 100644 index 000000000..a4a3921e2 --- /dev/null +++ b/docs/research/fixture-provenance/inventory.md @@ -0,0 +1,245 @@ +# Test-boundary inventory + +This inventory lists every boundary class the #1344 audit examined. A boundary +is a place where a test fixture, fake, stub or hand-built payload stands in for +something this repository does not control. For each boundary it records where +the fixture's shape came from, what the real producer returned, and how much +damage a wrong shape could do. The audit ran on `dev` at `35122105`. +[`index.md`](index.md) explains the method and keeps the divergence register. +[`captures.md`](captures.md) holds the observed shapes. + +Column meanings: + +- **Provenance**: `captured` (copied from a real response), `inferred` (written + from documentation, a sibling call, or what the code expects), `synthetic` + (deliberately artificial and labelled as such), or `real` (the test drives + the real producer). A qualifier narrows a label. `real, protocol-limited` + and `real, ASCII-only` mean the test drives the real producer with only part + of its inputs. `captured once` means one response was copied from a producer + whose output changes between requests. +- **Verdict**: `match`, `match in effect` (the fake differs from the real shape + only in a way the consumer already handles), `diverges`, `unverified` (no + real response could be obtained), or `n/a` (nothing consumes the shape). +- **Blast**: `H` for a security control, an authorization or data-integrity + path, or a rarely exercised automated path; `M` for a path that fails closed + but blocks real work; `L` for a hot path that fails loudly. + +Paths are relative to the repository root. `T/` is `implementations/python/tests/` +and `P/` is `implementations/python/packages/`. + +## GitHub platform and workflow runtime + +67 hand-built fixtures, `gh` stubs and assumed platform behaviors in 16 test +modules, plus about 25 `github.*` and default-environment reads across the 11 +workflows. None of the fixtures records its source. + +| Boundary | Consumer | Fixture | Provenance | Real observation | Verdict | Blast | +|---|---|---|---|---|---|---| +| `pull_request.body` on a PR without a description | `tools/check_pr_body.py:467-469` | `T/test_pr_body_guard.py:633`, `T/test_pr_body_policy_migration.py:61` | inferred | `"body": null` (REST, PR #25); webhook schema allows `["string","null"]` | diverges (GH-1) | L | +| PR author and head ref for the automation exemption | `tools/check_pr_body.py:190-200` | `T/test_pr_body_guard.py:524-541` | inferred | Dependabot head `dependabot/github_actions/dev/...`; release PRs target `main` | match | M | +| PR title and optional body for the title lint | `tools/check_pr_title.py:259-280` | `T/test_pr_title_guard.py:138-150`, `:271-288` | inferred | title is a string; body is a string or null and is handled | match | L | +| Issues REST read for the closing-route check | `tools/pr_body_issue_scope.py:69-101` | `T/test_pr_body_guard.py:199-217`, `:374-384`, `:561-588` | inferred | open issue without `pull_request`; PR-backed issue with `pull_request`; `"body": null` | match | M | +| Issues REST 301 for a transferred issue | `tools/pr_body_issue_scope.py:79-86` | none | none | no transferred issue available to capture | unverified | L | +| Closing keywords in PR bodies that target `dev` | `tools/check_pr_body.py:164-187`, `:296-314` | `T/test_pr_body_guard.py:228-317` | inferred | default branch is `main`; PR #1413 (base `dev`) says `Closes #1361` and has `closingIssuesReferences: []` | diverges (GH-2) | M | +| `gh api git/blobs --jq .content` | `.github/workflows/pr-body-policy.yml:47-58` | `T/test_pr_body_policy_migration.py:40-56` | inferred | base64 wrapped at 60 columns; decodes to the pinned digests | match | H | +| `gh api pulls/N/commits --jq` | `.github/workflows/pr-title-lint.yml:57-63` | `T/test_pr_title_guard.py:239-262` | inferred | raw multi-line text | match | L | +| `gh release view --json databaseId,isDraft,tagName` | `.github/workflows/release-please.yml` | `T/test_release_workflows.py:84-100` | inferred | `{"databaseId":401554295,"isDraft":true,"tagName":"v6.0.1"}` | match | H | +| `gh release view --json assets` | `.github/workflows/release-please.yml:1070-1080` | `T/test_release_workflows.py:186-193` | inferred | `.assets[].name` present; assets also carry `state` and `digest` | match | M | +| `gh release upload` without a checkout | `.github/workflows/release-please.yml:1104` | `T/test_release_workflows.py:202-215` | inferred | run 36967479436: `failed to run git: fatal: not a git repository` | diverges (GH-3) | H | +| Tag ref and annotated-tag dereference | `.github/workflows/release-please.yml:746-791` | `T/test_release_workflows.py:301-306`, `:801-835` | inferred | lightweight tag `type: commit`; annotated tag `type: tag`, then commit | match | H | +| PATCH `releases/{id}` response | `.github/workflows/release-please.yml:1156-1170` | `T/test_release_workflows.py:137` | inferred | `id` is a number and `draft` a boolean (GET) | match | M | +| release-please action outputs | `.github/workflows/release-please.yml:47-99` | `T/test_release_workflows.py:1238-1263` | inferred | `RELEASE_PLEASE_TAG: v6.0.1`, 40-hex SHA | match | H | +| `gh attestation verify --format json` identity fields | `tools/release_evidence_verifier.py:115-143` | `T/test_issue_1226_attestation_verifier.py:24-38` | inferred | same fields and values in the real v6.0.1 output | match | H | +| Number of verified attestations per artifact | `tools/release_evidence_verifier.py:107-111` | `T/test_issue_1226_attestation_verifier.py:99-103` | inferred | gh 2.101.0 returns every verified attestation for a digest | diverges (GH-4) | H | +| `GITHUB_RUN_ATTEMPT` across the jobs of one run | `tools/release_evidence_admission.py:398-408` | `T/test_issue_1226_release_admission.py:283-287` | inferred | run 36342459179: carried-over jobs started in attempt 1, re-run jobs in attempt 2 | diverges (GH-5) | M | +| `GITHUB_SHA` and `GITHUB_WORKFLOW_SHA` | pass-through, `tools/release_evidence.py:103-110` | `T/test_issue_1226_release_evidence_cli.py:56-70` | synthetic | both are `3d59d0eb...` in the v6.0.1 evidence index; the fixture's distinct values deliberately check that the two fields map separately (`docs/decisions/package-artifacts/issue-1226-preflight.md:84-85`) | match | L | +| `GITHUB_SHA` on `workflow_dispatch` | `tools/release_evidence.py:105`, `:449` | same as above | inferred | every listed release run was a `push` run | unverified | M | +| Runner image environment | `tools/bootstrap_profile.py:652-671` | `T/test_issue_1217_bootstrap_profiles.py:548-577` | inferred | `ImageOS` `ubuntu24`; image versions match the documented format | match | L | +| `GITHUB_HEAD_REF` and `GITHUB_BASE_REF` | `tools/check_requirement_governance.py:147-148` | `T/test_requirement_governance.py:192-203` | inferred | PR #1409: head `dev`, base `main`; empty on push | match | L | +| Actions runs and jobs JSON | `tools/ci_latency_report.py:129-141` | `T/test_ci_latency_report.py:12-123` | inferred | `run_started_at` resets on re-run | diverges (GH-7) | L | +| Workflow event fields | the 11 workflows | `T/test_pr_body_guard.py:715-727`, `T/test_issue_1313_workflow_policy.py:61-90` | inferred | `merged` and `merge_commit_sha` filled on close; Phase E run 37161479661 has `pull_requests: []` and the workflow carries the PR number in `run-name` | match | H | +| Release asset `state` | `.github/workflows/release-please.yml:1078-1102` | `T/test_release_workflows.py:186-193` | inferred | OpenAPI enum `uploaded`, `open`; an `open` asset cannot be produced read-only | unverified | L | + +## libvirt Python API + +98 fake definitions: 71 fake methods on fake connections and objects, 2 +`sys.modules["libvirt"]` error stubs and 25 inline stubs. They reduce to 16 faked +API surfaces. The observations come from libvirt 12.7.0 (`test:///default`) with +production-rendered XML, unless the row says otherwise. + +| Boundary | Consumer | Fixture | Provenance | Real observation | Verdict | Blast | +|---|---|---|---|---|---|---| +| `libvirtError` identity and `get_error_code()` | `P/raes_backend_libvirt/drivers/libvirt/_native.py:61-87` | `T/test_libvirt_backend_driver.py:36-51` | inferred | `libvirt.libvirtError`, not a `KeyError`, integer code | match | L | +| Missing domain or network, generic-driver fakes | `P/raes_backend_libvirt/drivers/libvirt/_native.py:208-226` | `T/test_libvirt_backend_driver.py:127-137` | inferred | code 42 `Domain not found`; code 43 `Network not found: ...` | match | H | +| Missing domain or network, TechVault-family fakes | `P/raes_backend_libvirt/_techvault_native_ops.py:48-62`, `P/raes_backend_libvirt/techvault_lifecycle.py:70-90` | `T/test_libvirt_backend_techvault_native.py:105-109` and two sibling modules | inferred | code 42 or 43, never `KeyError` | diverges (LV-2) | M | +| Missing nwfilter | `P/raes_backend_libvirt/drivers/libvirt/deployment.py:397`, `:442` | `T/test_libvirt_backend_driver.py:105-109` | inferred | libvirt's `nwfilterLookupByName` (`src/nwfilter/nwfilter_driver.c`) raises `VIR_ERR_NO_NWFILTER` (62); the test driver has no nwfilter driver | unverified | H | +| `defineXML` and `networkDefineXML` identity | `P/raes_backend_libvirt/drivers/libvirt/deployment.py:428-430`, `:464-490` | `T/test_libvirt_backend_driver.py:111-125` | inferred | a name or UUID collision raises code 9; an owned object can live under another name | diverges (LV-1) | H | +| `nwfilterDefineXML` | `P/raes_backend_libvirt/drivers/libvirt/deployment.py:400-404` | `T/test_libvirt_backend_driver.py:99-103` | inferred | libvirt's `virNWFilterObjListAssignDef` (`src/conf/virnwfilterobj.c`) rejects a name or UUID mismatch with code 9 and updates in place only when both match; the test driver has no nwfilter driver | unverified | L | +| `create()` | `P/raes_backend_libvirt/drivers/libvirt/deployment.py:222` | `T/test_libvirt_backend_driver.py:68-71` | inferred | `create()` on a running domain raises code 1; production catches any exception | match | L | +| `destroy()` on an inactive object | `P/raes_backend_libvirt/drivers/libvirt/_native.py:188-205` | `T/test_libvirt_backend_driver.py:73-76` | inferred | code 55 `domain is not running`; production tolerates 55 | diverges (LV-4) | L | +| `undefine()` | `P/raes_backend_libvirt/drivers/libvirt/deployment.py:434`, `:455`, `:482` | `T/test_libvirt_backend_driver.py:78-79` | inferred | a running persistent domain becomes transient and stays listed; refusals (managed save, snapshots, NVRAM) are not modeled | diverges (LV-4) | M | +| `isActive()` | `P/raes_backend_libvirt/drivers/libvirt/deployment.py:149-150` | every fake | inferred | `0` and `1` as integers | match | L | +| `UUIDString()` | `P/raes_backend_libvirt/drivers/libvirt/_native.py:134-148` | `T/test_libvirt_backend_techvault_native.py:67-70` returns `None` | inferred | always a lowercase string | match in effect | L | +| `name()` | `P/raes_backend_libvirt/techvault_lifecycle.py:151-160` | TechVault fakes | inferred | string | match | L | +| `XMLDesc(0)` | `P/raes_backend_libvirt/techvault_observation.py:171-198` | three TechVault-family modules | inferred | normalized readback: `131072`, `currentMemory`, `` | diverges (LV-3) | M | +| `listAllDomains()` and `listAllNetworks()` | `P/raes_backend_libvirt/techvault_lifecycle.py:139-148` | TechVault fakes | inferred | defined but inactive objects are listed | match | L | +| `get_error_domain()` and `get_error_level()` | none | `T/test_libvirt_failure_observability.py:47-51` | inferred | not consumed | n/a | - | +| `libvirt.open()` | `P/raes_backend_libvirt/drivers/libvirt/_native.py:158-160` | not faked (connection injected) | - | a bad URI raises code 38; never returns `None` | match | L | + +Committed real-daemon evidence covers less than its names suggest: +`tools/real-daemon/libvirt_smoke.py` has no committed output. A copy run against +`test:///default` with only the URI changed passed 3 of 12 checks, because the +harness predates the renames in #736 and #730 and the absence classification in +#1191. `tools/real-daemon/evidence/guest-certified-asr519-20260712T031842Z.json` +was string-edited after capture in `13f67129`: the retired project name in its +resource names and guest paths was replaced with `raes`, so names such as +`raes-evidence-guest-vm` were never reported by a daemon. The edit left the shape +fields unchanged. + +## Guest-side libvirt appliance + +31 fixtures in 9 test modules plus `T/libvirt_interface_fixtures.py`, checked +against six boots of the production-built appliance under `qemu-system-x86_64` +(Ubuntu noble `busybox-static` 1.36.1, kernel `linux-image-6.8.0-1067-aws`), +`virsh test:///default`, `qemu-img`, PyYAML 6.0.3 and upstream cloud-init and +libvirt source. + +| Boundary | Consumer | Fixture | Provenance | Real observation | Verdict | Blast | +|---|---|---|---|---|---|---| +| Guest `MemTotal` | `P/raes_backend_libvirt/guest_appliance.py:219`, `P/raes_backend_libvirt/guest_observation.py:65`, `:245-246` | `T/test_libvirt_backend_guest_certified.py:115`, `:490`, `:683` | inferred | `memory_mib 81` at 128 MiB; committed evidence says 78; 207 at 256 MiB | diverges (GU-1) | H | +| Guest account `disabled` report | `P/raes_backend_libvirt/guest_appliance.py:177-178`, `:246` | `T/test_libvirt_backend_guest_certified.py:183`, `:600`, `:608` | inferred | every account reports `1`; enabled accounts are written with `*` | diverges (GU-2) | H | +| Serial fact channel line endings | `P/raes_backend_libvirt/guest_transport.py:86-98` | `T/test_libvirt_backend_guest_certified.py:141` | inferred | raw bytes end in `\r\n`; `splitlines()` handles them | match in effect | L | +| Other fact lines (interfaces, content, services) | `P/raes_backend_libvirt/guest_appliance.py:213-259` | `T/test_libvirt_backend_guest_certified.py:108-141` | inferred | same format in the boot capture | match | - | +| Fact-file ownership under libvirtd | `P/raes_backend_libvirt/guest_transport.py:75-83` | stub transport, never reads a file | inferred | libvirt source creates the file root-owned `0600`; no live daemon was available | unverified | M | +| Initramfs archive | `P/raes_backend_libvirt/_initramfs.py:128-160` | `T/test_libvirt_boot_artifacts.py:68`, `:200-218` | synthetic | the kernel unpacked the production archive and ran `/init` | match | - | +| Static BusyBox ELF preflight | `P/raes_backend_libvirt/_initramfs.py:216-284` | `T/test_libvirt_boot_artifacts.py:26-56` | synthetic | real static, PIE, arm64 and Mach-O binaries classify as expected | match | - | +| `raes.challenge` kernel argument | `P/raes_backend_libvirt/techvault_matrix.py:232-236` | `T/test_libvirt_backend_guest_certified.py:30` | inferred | round-trips in the boot | match | - | +| cloud-init `user-data` parsing | `P/raes_backend_libvirt/cloudinit.py:106-121` | `T/test_libvirt_backend_cloudinit.py:28-31` uses `json.loads` as the oracle | inferred | cloud-init uses `yaml.safe_load`; astral-plane characters come back as lone surrogates | diverges (GU-3) | M | +| NoCloud `meta-data` | `P/raes_backend_libvirt/cloudinit.py:124-140` | `T/test_libvirt_backend_cloudinit.py:113-135` | inferred | PyYAML parses it to a mapping | match | - | +| `genisoimage` argv and exit code | `P/raes_backend_libvirt/drivers/seed.py:111-133` | `T/test_libvirt_backend_driver.py:211-218` | inferred | argv matches upstream NoCloud documentation; tool not available locally | unverified | L | +| Committed guest evidence artifact | `P/raes_operations/_evidence_run_native.py:99-141` | `T/test_libvirt_backend_guest_certified.py:747-773`, `T/test_libvirt_evidence_run.py:61-74` | synthetic | the committed artifact fails the current validator with 2 problems; no test loads it | diverges (GU-4) | M | + +## Network services and supply chain + +140 fixtures or hand-built payloads (about 190 parametrized cases) across curl +acquisition, bootstrap, vocabulary snapshots, the module registry, OCI images, +release evidence and archives. The audit also re-checked 52 locked artifact +values against their producers. A further 31 fixture builders and 30 lock values +cover the Isabelle and generic-tool archives. + +| Boundary | Consumer | Fixture | Provenance | Real observation | Verdict | Blast | +|---|---|---|---|---|---|---| +| curl exit codes and retries over HTTP/2 | `tools/maintained_client_acquisition.py:146-151`, `:236-243`; `tools/bootstrap_profile.py:70-71` | `T/test_issue_1217_bootstrap_profiles.py:776-819` (`http.server`, HTTP/1.x only) | real, protocol-limited | macOS curl 8.7.1: oversize transfer and HTTP 503 both exit 56, no retry; with `--http1.1`: 63 and 22 after 3 attempts; Ubuntu 24.04 curl 8.5.0 returns 63 on both protocols | diverges (NS-1) | M | +| `curl --version` text | `tools/maintained_client_acquisition.py:85-89` | `T/test_issue_1137_maintained_client_acquisition.py:86`, `:138` | inferred | parsed correctly | match | L | +| TLS rejection exit codes | `tools/maintained_client_acquisition.py:26`, `:238` | `T/test_issue_1217_bootstrap_profiles.py:900-928` | real | self-signed, expired, wrong-host and untrusted-root endpoints exit 60 | match | H | +| Redirects and unknown-length bodies | `tools/maintained_client_acquisition.py:139-157` | `_CurlFixture` routes | real | GitHub 302 to a signed CDN URL is followed; unknown length is bounded | match | H | +| NIST CSF export byte identity | `tools/check_nist_csf_defensive_vocabulary.py:219-224` | `T/test_issue_1221_*.py:70-75`, `:129-151` | captured once | two downloads gave 107517 and 107518 bytes with different digests; neither matches the lock | diverges (NS-4) | L | +| ATT&CK, ATLAS and W3C snapshots | `acquire_locked_bytes` | `T/test_issue_1221_*.py` | captured | digests and sizes equal the lock | match | L | +| uv, CPython, actions/python-versions and sonar-scanner archives | `tools/bootstrap_profile.py:445-507` | lock | captured | producer digests equal the lock | match | H | +| conftest, gitleaks, vale and osv-scanner release assets | `tools/verified_tool_installation.py:160-228` | `T/test_issue_1137_*.py:383-395`, `T/test_issue_1219_*.py:35-207` | inferred | all 12 assets equal the lock and install through the production code | match | H | +| Isabelle archive | `tools/verified_tree_archive.py:280-362` | `T/test_issue_1220_isabelle_acquisition.py:40-79` | inferred | both mirrors serve the locked size; CI admits the real archive | match | H | +| OCI distribution API for `oci:` imports | `P/raes/module_registry/__init__.py:156-186` | `T/test_sdl_module_registry.py:96-170` | inferred | ghcr.io, Docker Hub and ECR Public answer `401` with a `Bearer` challenge; blobs redirect with `307` | diverges (NS-2) | M | +| Docker Hub alpine image graph | `tools/oci_release_image.py:115-181` | lock | captured | digests, sizes and `diff_ids` are byte-exact | match | H | +| `docker image inspect` output | `tools/oci_release_image.py:54` | `T/test_oci_release_image.py:68-83` | inferred | CI run 37270742395 passed against a real daemon | match | H | +| `docker run` and `docker container inspect` | `P/raes_reference_backend/drivers/oci.py:288-367` | `T/test_reference_backend_oci_driver.py:19-50` | inferred | ID on stdout, pull progress on stderr | match | M | +| OCI layer list | `tools/tooling_artifact_policy_oci.py:102-108` | `T/test_tooling_artifact_policy.py:1584-1587` | inferred | the linux/amd64 manifest of the Docker Official Image `tomcat:latest` (`sha256:594377d0...`) lists one empty layer three times | diverges (NS-5) | L | +| Lock-shaped image selection fixture | no current consumer | `T/test_oci_release_image.py:122-124`, `:154-155` | inferred | the real lock names the platform manifest; arm64 has variant `v8` | n/a (NS-6) | L | +| gh output with no attestation | `tools/release_evidence.py:292-307` | `T/test_issue_1226_attestation_verifier.py:74-77` | inferred | exit 1, empty stdout, stderr `Error: HTTP 404` | diverges (NS-7) | L | +| Release build inventory subjects | `tools/release_evidence_admission.py:314-320` | `T/test_issue_1226_release_admission.py:70`, `:172`; `T/test_issue_1227_release_publication.py:64` | inferred | the real inventory has 3 subjects; the sdist-built test wheel equals the wheel | diverges (NS-3) | M | +| CycloneDX 1.6 SBOM | `tools/release_evidence_documents.py:75-150` | `T/test_issue_1226_evidence_documents.py:24-107` | inferred | real SBOMs validate with 0 errors | match | L | +| PyPI per-version JSON | `.github/workflows/release-please.yml:809-848` | `T/test_release_workflows.py:1378-1497` | inferred | digests present; 404 body `{"message": "Not Found"}` | match | H | +| curl failure output in the PyPI check | `.github/workflows/release-please.yml:809-822` | `T/test_release_workflows.py:1394-1411` | inferred | curl prints `000` and exits 6; the stub exits 0 | diverges (NS-8) | L | +| AWS EC2 CLI JSON | `tools/real-daemon/run_aws_smoke.sh:70-146` | none | none | not captured | unverified | M | +| Lock digests for actions that download their own tools | scorecard and sonar actions | lock | captured | the consuming actions pull by tag or check a signature, so the lock digest is not enforced | unverified | M | + +## Subprocesses and command-line tools + +The audit drove each tool with the production argv in the states the consumer +handles. Fixture counts were not recorded for this class. + +| Boundary | Consumer | Fixture | Provenance | Real observation | Verdict | Blast | +|---|---|---|---|---|---|---| +| `conftest test --output json` | `tools/policy/conftest_tool.py:59-106` | `T/test_repo_policy_tools.py:1288-1316` (failure output only) | inferred | load and evaluation errors exit 1 with an empty stdout, which the runner read as no failures | diverges (CL-1) | H | +| `conftest verify` | `tools/policy/conftest_tool.py:109-125` | `T/test_repo_policy_tools.py:486` (command wiring) | inferred | exits 1 on a parse error, 0 on a policy whose evaluation conflicts only on real input | match | M | +| gitleaks directory scan over a symlink farm | `tools/nox_support/runner.py:361-380` | `T/test_issue_1350_fixture_secret_scan.py:19-83` | real | the production argv reports leaks through `--follow-symlinks`; exit 1 with findings | match | H | +| osv-scanner exit codes | `tools/osv_scanner_tool.py:17-33` | `T/test_repo_policy_tools.py:3233-3266` | inferred | 0 clean, 1 findings, 127 network failure, 128 no packages | match | M | +| vale exit codes | docs nox lane | none | - | 0 clean, 1 alerts, 2 runtime error; both non-zero codes fail the lane | match | L | +| git path listings without `-z` | `tools/nox_support/runner.py` (`_changed_paths`, `_tracked_repo_paths`), `tools/policy/common.py` (`changed_paths`) | scratch repositories with ASCII names | real, ASCII-only | git quotes non-ASCII and control-character paths; the runner drops them and the policy helper returns the quoted string | diverges (CL-2) | M | +| `docker image inspect` and `docker run` | see the network table | - | - | Docker 29.7.2 output matches the fixtures | match | M | +| Isabelle build output and `bwrap` errors | `tools/isabelle_tool.py:345-349`, `:414-422` | `T/test_issue_963_participant_opacity_proof.py:227-241` | inferred | upstream sources print `bwrap: ` and `Finished `; CI replay passes | match | M | +| `uv build --sdist` file name | `tools/python_closure.py` | fake writes `raes-1.tar.gz` | inferred | the distribution name is `raes`, so `raes-.tar.gz` | match | L | +| pytest JUnit XML for the docker lane | release workflow skip check | string assertions only | inferred | real JUnit marks skips with `` as the parser expects | match | M | +| coverage.py JSON | `tools/nox_support/test_lanes.py` | coverage fixtures | inferred | coverage.py 7.13.5 JSON has `totals.covered_lines` and `totals.num_statements` | match | L | +| `check-jsonschema` | schema checks | - | - | not captured before the audit stopped | unverified | L | + +## Control-plane store and the requirements client + +The observations come from probes that run the store's SQL and the production +`LocalControlPlaneStore` against SQLite 3.50.4 under CPython 3.14.4 on macOS. + +| Boundary | Consumer | Fixture | Provenance | Real observation | Verdict | Blast | +|---|---|---|---|---|---|---| +| Driver shapes: `BLOB` read-back, unique and primary-key violations, nested `BEGIN IMMEDIATE`, `SQLITE_BUSY` | `P/raes_runtime/control_plane_store_local*.py` | store tests | real | errors and return values as the store expects | match | H | +| WAL sidecars, backup journal mode, VFS fallbacks | `P/raes_runtime/control_plane_store_maintenance.py` | store tests | real | WAL persists through backup; non-database files raise `DatabaseError` | match | M | +| `PRAGMA quick_check` on a corrupt file | store admission | `_QuickCheckFailureConnection` fake (`T/test_issue_1092_control_plane_crash_consistency.py:1028`) returns a non-`ok` row | inferred | index and page damage return non-`ok` rows and admission refuses; a damaged table root or truncation raises `sqlite3.DatabaseError` from the PRAGMA itself | diverges (ST-3) | L | +| Runtime-owner lease across `fork()` | `P/raes_runtime/control_plane_store_lease.py:217-231` | `T/test_issue_1092_control_plane_crash_consistency.py:2432` (`test_runtime_owner_lease_rejects_and_closes_in_a_different_process_identity`) patches `os.getpid` in one process | inferred | a forked child that calls `LOCK_UN` on its inherited descriptor releases the parent's lock; the simulated fork cannot observe that | diverges (ST-1) | H | +| Commit failure | `P/raes_runtime/control_plane_store_local_codec.py:13-24` | `T/test_run_310_supervisory_lifecycle.py:872`, `T/test_runtime_control_plane_api.py:1878` raise `OSError("commit failed")` inside the transaction body | inferred | a real commit failure raises `sqlite3` errors from `commit()`; the per-operation connection close discards it, so nothing persists | diverges (ST-2) | L | +| Ground Control HTTP client transport | `tools/policy/requirement_governance.py` | `T/test_requirement_governance.py:150-152` fakes `HTTPError`, `URLError` and `TimeoutError` | inferred | a dropped connection, a truncated body, an HTML login page and non-UTF-8 JSON raise exceptions that are not `GroundControlError` | diverges (ST-4) | M | +| Store migration across releases and request-commitment drift | store migration code | store tests | - | probes were written but recorded no verdict | unverified | H | + +## MCP, filesystem, time and internal stand-ins + +| Boundary | Consumer | Fixture | Provenance | Real observation | Verdict | Blast | +|---|---|---|---|---|---|---| +| MCP stdio wire: initialize, tools/list, tools/call | `P/raes_mcp` | MCP tool tests call the functions directly | inferred | protocol `2025-11-25`; all 31 tools declare an output schema; structured and text content agree | match | M | +| Rejected MCP arguments | `P/raes_mcp` | `T/test_experiment_authoring.py:195-206` passes `spec_content` as a string | inferred | a client that sends an object gets the framework's validation error, which includes `input_value` and, for small inputs, the value | diverges (TM-3) | L | +| Filesystem errors (`flock`, `O_NOFOLLOW`, `O_DIRECTORY`, `fsync` on directories) | store paths and lease | store and installation tests | real | macOS error classes and errnos captured | match | M | +| Workflow step timestamps for compensation order | `P/raes_runtime/control_plane_workflows.py:52` | `T/test_runtime_control_plane_api.py:2148`, `:2277` have one compensable step | inferred | producers drop the fraction at a whole second (`...:01Z` then `...:01.001000Z`), so the string sort reverses them | diverges (TM-1) | H | +| Runtime-fact `requested_at` | `P/raes_runtime/runtime_fact_dispatch.py:36`, `P/raes_runtime/runtime_fact_binding_policy.py:119-124` | `T/test_runtime_fact_bindings.py:107`, `:233-243` use `Z` timestamps | inferred | a naive or date-only value is admitted, then binding raises `TypeError` | diverges (TM-2) | M | +| Unavailable container runtime | `P/raes_reference_backend/drivers/oci.py` | driver tests | inferred | a stopped Docker daemon gives `reference-backend.driver.command-failed` | match | L | +| LilRAE `aptl-evidence-bundle/v1` and `aptl.run-record/v2` exports | cross-backend corpus consumer | hand-built exports in `T/test_cross_backend_corpus.py:109-177` | inferred | both exports, as LilRAE's producer source at `a5833df9` builds them, pass the corpus checks | match | M | + +## Backend manifests and realization envelopes + +52 fixtures: 37 corpus files and 15 in-test constructions, compared with the +stub, reference, libvirt and LilRAE APTL producers (LilRAE at `a5833df9`, read +only). + +| Boundary | Consumer | Fixture | Provenance | Real observation | Verdict | Blast | +|---|---|---|---|---|---|---| +| Stub backend manifest | conformance validators | `contracts/fixtures/.../backend-manifest-v2/valid/stub.json` | captured | 116 of 116 paths equal the producer apart from the version | match | L | +| Reference processor manifest | `ProcessorManifestV2Model` | `contracts/fixtures/.../processor-manifest-v2/valid/reference.json` | captured | 14 of 14 paths equal apart from the version | match | L | +| Processor and backend mutual compatibility | `P/raes_processor/trial_realization.py:224-228`, `P/raes_processor/trial_compiler/apparatus.py:389-401` | `T/test_sce_002_trial_realization.py:67` and `T/test_runtime_contracts.py:81` patch `compatibility.backends`; `:91` renames the backend to match | inferred | the real processor declares `["stub"]` only; every non-stub backend is rejected | diverges (MF-1) | M | +| Mixed-runtime manifest anti-substitution check | `P/raes_runtime/mixed_runtime.py:212-214` | `T/test_issue_1016_mixed_runtime_coordination.py:278-291` | inferred | the APTL manifest keeps authored OS order and a list where the round trip sorts and uses a tuple, so the admitted manifest is rejected | diverges (MF-2) | M | +| Capture offers | `P/raes_processor/capture_admission.py:212-218` | `T/test_sce_002_trial_compiler.py:361-385` | inferred | no producer emits an offer | unverified | M | +| Negative backend-manifest corpus | `T/test_backend_manifest.py:1179-1183` | 9 files under `backend-manifest-v2/invalid/` | inferred | each file fails for its named defect and for incidental ones | diverges (MF-3) | L | +| Plan inspection with `--manifest` | `P/raes_cli/processor.py:136-145` | `T/test_plan_inspection_cli.py:88-190` | captured | envelope-bearing manifests are refused, as documented | match | L | +| Published realization envelopes | envelope loaders | `contracts/realization-envelopes/*` | captured | producer output equals the files | match | L | +| APTL manifest admission | conformance and service materialization | negative paths only | - | the real APTL manifest is admitted | match | L | + +## Runtime snapshots and control-plane payloads + +| Boundary | Consumer | Fixture | Provenance | Real observation | Verdict | Blast | +|---|---|---|---|---|---|---| +| Receipts, statuses and snapshot carriers | conformance and contracts | `contracts/fixtures/` runtime fixtures | captured | real payloads carry the same keys | match | M | +| Snapshot validated by target conformance | conformance hermetic projection | conformance tests | inferred | the projection omits `realization_envelope` and `realization_provenance`, which the `/snapshot` API publishes | diverges (RS-1) | M | +| Public snapshot used by the SEM-222 test | `/snapshot` route | the test serializes with the durable store | inferred | the public route drops episode closure records and mixed-composition state that the store keeps | diverges (RS-2) | M | +| Participant behavior history in a real snapshot | conformance semantics | corpus adds `participant.*` entries with empty payloads | inferred | a `/snapshot` from the real control plane with behavior history has no `participant.behavior` entry and fails semantic conformance | diverges (RS-3) | M | +| LilRAE snapshot projection | `RuntimeSnapshotEnvelopeModel`, conformance | - | - | a projection as LilRAE's producer source at `a5833df9` builds it is accepted with 0 diagnostics | match | L | + +## Release, conformance and experiment evidence + +29 fixtures across 10 boundaries. The table lists seven. The other three match: +the release evidence index and SBOM subject hash, the optional `task_ref` and +`apparatus_context_ref` checks on evidence records, and the hand-authored +lineage ledger example. + +| Boundary | Consumer | Fixture | Provenance | Real observation | Verdict | Blast | +|---|---|---|---|---|---|---| +| Release build inventory | `tools/release_evidence_admission.py:314-316` | see NS-3 | inferred | 3 subjects; derived wheel equals the wheel | diverges (NS-3) | M | +| Evidence record `capture_spec_ref` | `P/raes_contracts/evidence_satisfaction.py:98-107` | all 5 files under `contracts/fixtures/experiment-core/experiment-evidence-record-v1/valid/` | inferred | LilRAE's builder omits `ref_version`; the schema accepts the record and the validator rejects it | diverges (EV-1) | M | +| libvirt evidence-run artifact | cross-backend corpus | the test drives the real producer | real | descriptor `{libvirt-qemu, 5.0.0}` | match | L | +| Conformance report | release workflow | `T/test_backend_conformance_cli.py:43-63` | captured | real producers pass the same case validator | match | L | +| Minimal plans | operation routes | `contracts/fixtures/.../plans/*/valid/minimal.json` | inferred | a real plan posted to the API returns 200 | match | L | +| Trial compiler identity vectors | `T/test_sce_002_trial_compiler.py:748-758` | `identity-vectors.json` | captured | golden values from the real function | match | L | +| Formal-evidence archives | `tools/formal_semantic_validation` | `docs/research/formal-semantic-validation/evidence/` | captured | pinned and replay-checked | match | L |