P5.3: the backend conformance suite, specced and planned - #89
Merged
Conversation
Design for the backend conformance suite, roadmap step P5.3. Covers the first of P5's two exit clauses only; the product-level `gascan-e2e`-on-arca work is named as out of scope rather than left implicit. **The roadmap's "extract the conformance suite from `fake_runtime.rs`" reads as though one must be written, and it must not.** `crates/gascan-core/tests/backend_contract.rs:149` is already `pub async fn backend_contract(backend: &dyn RuntimeBackend)`, a generic walk over the trait object. `crates/gascan-apple/tests/live/backend_contract.rs` hand-rolls 65 lines over the same ground because `backend_contract` sits in a `tests/` target, which is not a library, so `gascan-apple` cannot import it. The duplication is a consequence of location. So the task is to make the contract importable, grow it, and instantiate it three times. **Where it lives was decided by a lint, not a preference.** `crates/gascan-core/src/lib.rs:2` denies `clippy::unwrap_used` and CI enforces it at `.github/workflows/ci.yml:54`. MEASURED at `10e3342`: `fake_runtime.rs` has 0 `unwrap()` and no `#[allow]`, meeting that bar honestly; `backend_contract.rs` has 114. A new dev-dependency-only crate `gascan-conformance` keeps the denial intact and ships nowhere, where a module-level allow would weaken a production library to host test assertions. **An overbuild is recorded here rather than deleted, because the check that killed it generalises.** An earlier draft scoped an out-of-band mechanism for creating ownership-labelled resources so the `Foreign`/`Mismatched` assertions could run live. `classify_resource_ownership` is a pure function in shared code (`crates/gascan-core/src/runtime.rs:85-103`) and `crates/gascan-core/tests/resource_ownership.rs` already tests it exhaustively, so promoting those would run the same function three times. The rule extracted: an assertion earns promotion only if the behaviour it names is implemented separately by each backend. The one genuinely per-backend gap that survived is recorded as a follow-up, not folded in: classification is shared but *visibility* is not, and a backend that filters unlabelled resources would silently break drift detection at `crates/gascand/src/service.rs:3012`. The promotion estimate is a range, 6-8, and says so. An earlier draft said 13. Also recorded: the fake instantiation runs in CI every push, the arca one inherits the live tier CI already runs (`ci.yml:178`), and the apple one runs **nowhere** in CI -- no workflow step passes `--ignored` for `gascan-apple` or `gascan-e2e`. Any future claim that apple passes conformance has to say which machine and when. Three anchors in the first draft were wrong and were re-derived before commit: `apple_common/mod.rs` `command()` is at :961 not :964, `arca_common/mod.rs` at :303 not :308, and the START-HERE guard-mutation record is at :604.
Implements `docs/superpowers/specs/2026-08-20-backend-conformance-suite-design.md`. Eight tasks: create the crate, move the contract in and instantiate against the fake, strip the duplicate out of gascan-core, instantiate against apple and arca, promote start/stop idempotence, triage the remaining candidates, record the measurement. **Filling the plan's own holes is what found the blockers.** A first draft left three `/* ... */` markers in the arca task for the harness's constructor names, annotated as "read lifecycle.rs". Reading it surfaced three constraints that would each have failed the first run: - The manifest needs `user = 'root'`. The base layout is a stock alpine with no `workspace` account, and the engine hands `UserMode::Workspace` to `createContainer` as the literal string `workspace`, so `start` would fail on the image rather than on anything under test (`lifecycle.rs:24-30`). - The image needs a rewritten `Cmd`. Alpine's own is `/bin/sh`, which exits immediately with no tty; the contract does start -> exec -> stop and needs a container still running (`lifecycle.rs:35-37`). `layout_running` (`gascan-oci-fixture/src/lib.rs:39`) rewrites it without touching the rootfs. - The network must be `networked`, not the fixture's default `offline`. Offline is the capability the pinned engine does not honour, so an offline request would test the refuted property by accident. That third fact changed the crate's API: `for_image` takes a whole manifest rather than a network string, matching `policy_request_from_manifest(name, image, manifest)` at `live/common/mod.rs:718` -- `user` is a manifest fact and a network-only parameter cannot carry it. The tier already documents the reasoning: "The manifest is the only knob, deliberately." One deliberate change from the fixtures being moved: `RuntimeVersion::new(1, 0, 0)` becomes `(1, 1, 0)`. Verified safe -- `capabilities.version` reaches only an error message (`policy.rs:422`, inside `PolicyError::OfflineUnsupported`) and neither backend validates a request's version. 1.1.0 matches apple's floor (`probe.rs:36`) and the value arca's tier already uses. Task 5 step 6 has no expected result written for it, on purpose. Arca's run is the measurement the plan exists for, and step 7 requires committing the failing test with the failure quoted rather than weakening the contract.
PR #87 merged, so the cold-start block's headline was a decision already made. Replaced with the assignment the maintainer chose on 2026-08-20 -- P5.3, the backend conformance suite -- pointing at the design and the eight-task plan committed alongside this. The block carries the one fact that misleads a cold reader who skips the design: the roadmap's "extract the conformance suite from `fake_runtime.rs`" reads as an instruction to write one, and `crates/gascan-core/tests/backend_contract.rs:149` already **is** `pub async fn backend_contract(backend: &dyn RuntimeBackend)`. It is unreachable from the backend crates only because a `tests/` target is not a library, which is why `crates/gascan-apple/tests/live/backend_contract.rs` hand-rolls 65 lines over the same walk. Relocation and instantiation, not authorship. It also carries what the task is not -- the product-level `gascan-e2e`-on-arca work, P5.4/U5, offline -- because each is a plausible thing to widen into and each is named out of scope in the design with a reason. The flake and daemon-log queue is demoted from "the assignment" to what comes after it. Nothing in it changed.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Docs only — a design, an eight-task plan, and the handoff pointing at them. No code.
The finding that shaped both
The roadmap's P5.3 says "Extract the conformance suite from
fake_runtime.rs." That reads as though a suite must be written. It must not —crates/gascan-core/tests/backend_contract.rs:149is alreadypub async fn backend_contract(backend: &dyn RuntimeBackend).It is unreachable from
gascan-appleandgascan-arcabecause it lives in atests/target, which is not a library. That is whycrates/gascan-apple/tests/live/backend_contract.rshand-rolls 65 lines over the same walk. The task is relocation and instantiation, not authorship — materially smaller than the roadmap's wording implies.Where the shared code goes, decided by a lint rather than taste
crates/gascan-core/src/lib.rs:2deniesclippy::unwrap_used, enforced by CI at.github/workflows/ci.yml:54. MEASURED at10e3342:fake_runtime.rshas 0unwrap()and no#[allow], meeting that bar honestly;backend_contract.rshas 114. So a new dev-dependency-only crategascan-conformance— a module-level allow would weaken a production library to host test assertions, and aResultrewrite would stop assertion failures pointing at the assertion.An overbuild, recorded rather than deleted
An earlier draft scoped an out-of-band mechanism for creating ownership-labelled resources so
Foreign/Mismatchedcould be asserted live.classify_resource_ownershipturns out to be a pure function in shared code (gascan-core/src/runtime.rs:85-103) thattests/resource_ownership.rsalready tests exhaustively — promoting those would run the same function three times.The rule extracted, now applied to every promotion candidate: an assertion earns promotion only if the behaviour it names is implemented separately by each backend. The one genuinely per-backend gap that survived — classification is shared but visibility is not, so a backend filtering unlabelled resources would silently break drift detection at
gascand/src/service.rs:3012— is recorded as a follow-up, not folded in.Filling the plan's own holes is what found the blockers
A first draft left three
/* ... */markers in the arca task annotated "readlifecycle.rs". Reading it surfaced three constraints that would each have failed the first run:user = 'root'— the base layout is stock alpine with noworkspaceaccount, and the engine handsUserMode::WorkspacetocreateContaineras the literal string (lifecycle.rs:24-30).Cmd— alpine's own is/bin/sh, which exits immediately; the contract does start → exec → stop and needs a container still there (lifecycle.rs:35-37).networked, not the fixture defaultoffline— offline is the capability the pinned engine does not honour, so an offline request would test the refuted property by accident.That third fact changed the crate's API:
for_imagetakes a whole manifest rather than a network string, matchingpolicy_request_from_manifest(name, image, manifest)(live/common/mod.rs:718), becauseuseris a manifest fact. The tier already documents the reasoning: "The manifest is the only knob, deliberately."The criterion most likely to come under pressure
Task 5 step 6 — running the contract against arca for the first time — has no expected result written for it, on purpose. It is the measurement the whole task exists for. Step 7 requires committing the failing test with the failure quoted rather than weakening the contract. That is design acceptance criterion 8, and it is worth agreeing to before implementation starts rather than during it.
Also recorded
Where each instantiation runs is asymmetric: fake runs in CI every push, arca inherits the live tier CI already executes (
ci.yml:178), and apple runs nowhere in CI — no workflow step passes--ignoredforgascan-appleorgascan-e2e. Any future "apple passes conformance" claim has to name a machine and a date.