From 1f408539e3efbbd260ad721c85995a9719a1f895 Mon Sep 17 00:00:00 2001 From: Richard Kiene Date: Thu, 20 Aug 2026 11:01:42 -0700 Subject: [PATCH 1/3] docs: P5.3's design, and the suite it says to write already exists Design for the backend conformance suite, roadmap step P5.3. Covers the first of P5's two exit clauses only; the product-level `gascan-e2e`-on-arca work is named as out of scope rather than left implicit. **The roadmap's "extract the conformance suite from `fake_runtime.rs`" reads as though one must be written, and it must not.** `crates/gascan-core/tests/backend_contract.rs:149` is already `pub async fn backend_contract(backend: &dyn RuntimeBackend)`, a generic walk over the trait object. `crates/gascan-apple/tests/live/backend_contract.rs` hand-rolls 65 lines over the same ground because `backend_contract` sits in a `tests/` target, which is not a library, so `gascan-apple` cannot import it. The duplication is a consequence of location. So the task is to make the contract importable, grow it, and instantiate it three times. **Where it lives was decided by a lint, not a preference.** `crates/gascan-core/src/lib.rs:2` denies `clippy::unwrap_used` and CI enforces it at `.github/workflows/ci.yml:54`. MEASURED at `10e3342`: `fake_runtime.rs` has 0 `unwrap()` and no `#[allow]`, meeting that bar honestly; `backend_contract.rs` has 114. A new dev-dependency-only crate `gascan-conformance` keeps the denial intact and ships nowhere, where a module-level allow would weaken a production library to host test assertions. **An overbuild is recorded here rather than deleted, because the check that killed it generalises.** An earlier draft scoped an out-of-band mechanism for creating ownership-labelled resources so the `Foreign`/`Mismatched` assertions could run live. `classify_resource_ownership` is a pure function in shared code (`crates/gascan-core/src/runtime.rs:85-103`) and `crates/gascan-core/tests/resource_ownership.rs` already tests it exhaustively, so promoting those would run the same function three times. The rule extracted: an assertion earns promotion only if the behaviour it names is implemented separately by each backend. The one genuinely per-backend gap that survived is recorded as a follow-up, not folded in: classification is shared but *visibility* is not, and a backend that filters unlabelled resources would silently break drift detection at `crates/gascand/src/service.rs:3012`. The promotion estimate is a range, 6-8, and says so. An earlier draft said 13. Also recorded: the fake instantiation runs in CI every push, the arca one inherits the live tier CI already runs (`ci.yml:178`), and the apple one runs **nowhere** in CI -- no workflow step passes `--ignored` for `gascan-apple` or `gascan-e2e`. Any future claim that apple passes conformance has to say which machine and when. Three anchors in the first draft were wrong and were re-derived before commit: `apple_common/mod.rs` `command()` is at :961 not :964, `arca_common/mod.rs` at :303 not :308, and the START-HERE guard-mutation record is at :604. --- ...-08-20-backend-conformance-suite-design.md | 225 ++++++++++++++++++ 1 file changed, 225 insertions(+) create mode 100644 docs/superpowers/specs/2026-08-20-backend-conformance-suite-design.md diff --git a/docs/superpowers/specs/2026-08-20-backend-conformance-suite-design.md b/docs/superpowers/specs/2026-08-20-backend-conformance-suite-design.md new file mode 100644 index 0000000..b6390e1 --- /dev/null +++ b/docs/superpowers/specs/2026-08-20-backend-conformance-suite-design.md @@ -0,0 +1,225 @@ +# P5.3 — the backend conformance suite, design + +**Roadmap step:** P5.3 of `docs/superpowers/plans/2026-08-04-arca-integration-roadmap.md`, +worded there as *"Extract the conformance suite from `fake_runtime.rs`; run against fake, apple, +and arca backends."* P5's exit is *"`gascan-arca` passes conformance and existing `gascan-e2e`"*; +this design covers the **first clause only**. The second is named under "Out of scope". + +Every file:line and count below was re-derived on 2026-08-20 at `10e3342`. Re-derive rather than +trust them; this repository's durable docs have gone stale on their own numbers repeatedly. + +--- + +## 1. What is already there, because it changes the task + +The roadmap's "extract" reads as though a suite must be written. It must not. **The generic +conformance function already exists**, at `crates/gascan-core/tests/backend_contract.rs:149`: + +```rust +pub async fn backend_contract(backend: &dyn RuntimeBackend) +``` + +It takes a trait object and walks: `inspect` absent → `create` → assert a `Container` resource +among those created → `inspect` reports `Stopped` → `start` → `exec` `true` → assert +`ExecOutput::Exit { code: 0, signal: 0 }` → `stop` → `remove` → `inspect` absent. `FakeRuntime` +runs it through `fake_runtime_satisfies_backend_contract_through_trait_object`. + +**`crates/gascan-apple/tests/live/backend_contract.rs` does not call it.** It hand-rolls a +65-line walk over the same ground. That is not an oversight: `backend_contract` lives in a +`tests/` target, which is not a library, so `gascan-apple` **cannot** import it. The duplication +is a consequence of location, and location is what this design changes. + +So P5.3 is: **make the contract importable, grow it, and instantiate it three times.** + +## 2. Where the shared code lives, and why not `gascan-core` + +The obvious move is a `pub mod` in `gascan-core/src`, following `fake_runtime` — which is +unconditionally public at `crates/gascan-core/src/lib.rs:9`, establishing that this project will +ship test-support code in that library. + +**It does not work, and the reason is a hard gate rather than a preference.** +`crates/gascan-core/src/lib.rs:2` reads: + +```rust +#![deny(clippy::expect_used, clippy::panic, clippy::unwrap_used)] +``` + +CI enforces it — `cargo clippy --workspace --all-targets -- -D warnings`, `.github/workflows/ci.yml:54`. +MEASURED at `10e3342`: `fake_runtime.rs` contains **0** occurrences of `unwrap()` and carries **no** +`#[allow(clippy::…)]`; it meets that bar honestly. `backend_contract.rs` contains **114**. + +That leaves three options, and only one is honest: + +| option | cost | +|---|---| +| module-level `#[allow(clippy::unwrap_used)]` in `gascan-core/src` | weakens a *production* library's deliberate discipline to host test assertions | +| rewrite the contract to propagate `Result` | ~114 sites, and assertion failures stop pointing at the assertion | +| **a separate crate** | none of the above; the denial stays intact because the code is not in that crate | + +**Decision: a new crate, `gascan-conformance`.** It is a **dev-dependency only** of the crates that +run it, so unlike `fake_runtime` it is compiled into no shipped artifact at all. Panicking +assertions are correct in a test-support crate and wrong in `gascan-core`, and this puts each +where it belongs. + +**The fake instantiation lives inside `gascan-conformance` itself** (`tests/fake.rs`), taking +`gascan-core` as an ordinary dependency and reaching `gascan_core::fake_runtime::FakeRuntime`. +This is deliberate: putting it in `gascan-core/tests` instead would make `gascan-core` +dev-depend on a crate that depends on it. Cargo permits that cycle, but it is a cycle a reader has +to think about, and there is no reason to mint one. + +### Layout + +``` +crates/gascan-conformance/ + src/lib.rs the contract + the fixtures it needs + tests/fake.rs instantiation 1 — FakeRuntime, free, runs in CI +crates/gascan-apple/tests/live/backend_contract.rs instantiation 2 — replaces the 65-line duplicate +crates/gascan-arca/tests/live/conformance.rs instantiation 3 — new +crates/gascan-core/tests/backend_contract.rs keeps only what is genuinely fake-specific +``` + +The fixtures move too. `crates/gascan-core/tests/common/mod.rs` is **61 lines** and holds all three +the contract needs — `capabilities()` (`:27`), `create_request()` (`:40`), +`create_request_with_network()` (`:44`). + +## 3. What gets promoted, and what stays fake-only + +`backend_contract.rs` holds **23** `#[tokio::test]` functions. They are not one population. + +**Promote — assertions any backend must satisfy.** The lifecycle walk already in +`backend_contract()`, plus `duplicate_create_is_rejected_and_start_stop_are_idempotent`, +`exec_and_logs_preserve_binary_bytes_and_exact_exit_code`, +`exec_session_is_live_bidirectional_and_emits_one_exit`, +`create_collision_reports_resources_created_before_the_collision`, +`offline_fake_create_has_no_managed_network`, +`networked_fake_create_reports_network_then_volumes_then_container`, +`persistent_logs_are_isolated_by_exact_sandbox_id`. **Estimated 6–8 will promote cleanly**, and the +estimate is deliberately a range: each one has to be re-read to separate what any backend owes from +what this double happens to do. **An earlier draft of this analysis said 13. That was wrong** — see +§4 for the class of error it made. + +**Stays fake-only — tests of the double's controllability, not of the contract.** +`named_failure_is_injected_once_at_the_call_boundary`, +`every_backend_boundary_supports_fail_once_injection`, +`injected_post_mutation_create_failure_reports_partial_resources` (all `FailureBoundary` +injection); `literal_requests_are_recorded_in_order`, `fake_recreate_records_prepare_then_container_create`, +`removal_mutates_in_container_volume_network_order` (all assert through the fake's call recorder, +`calls()`/`outcomes()`, which no real backend has); `persistent_fake_runtime_reopens_runtime_truth_without_controller_state` +and `same_name_seeded_network_conflicts_without_adoption` (persistence and seeding). + +`removal_mutates_in_container_volume_network_order` is the interesting one: **the ordering it +asserts is real contract, but the instrument is not portable.** Against a real backend only the +*effect* is observable — the resources are gone — not the order they went in. It stays fake-only +and the residue is recorded here rather than papered over. + +**Neither — `cancellable_exec_session_cancel_is_idempotent` and +`cancellable_exec_session_drop_signals_backend`** test `ExecSession` itself, take no backend, and +simply stay where they are. + +## 4. What is NOT in this design, and the error that nearly put it here + +An earlier draft scoped a mechanism for creating ownership-labelled resources out-of-band, so the +`Foreign`/`Mismatched` assertions could run against real backends. **That was an overbuild, and the +check that killed it is worth recording** because the same reasoning applies to anything else +proposed for promotion. + +Ownership classification is **not per-backend**. `classify_resource_ownership` is a pure function +in shared code at `crates/gascan-core/src/runtime.rs:85-103`, taking `Option<&str>` and a +`SandboxLabel` — no backend involved. `crates/gascan-core/tests/resource_ownership.rs` already +tests it exhaustively across all three verdicts and every resource kind. Promoting those +assertions would run **the same function three times** and prove nothing new. The design note this +implements says as much: *"Ownership labels cross the wire; classification does not."* + +**The general rule this yields, and the one to apply to every promotion candidate in §3:** an +assertion earns promotion only if the behaviour it names is *implemented separately by each +backend*. Shared code tested once is done. + +### The one genuinely per-backend gap, recorded as a follow-up + +Classification is shared; **visibility is not**. If `gascan-apple`'s listing parse drops unlabelled +entries, or the Arca engine filters them server-side, then `list_resources` returns a clean +inventory and gascand's drift detection never emits `ReconcileFinding::UnknownUnowned` +(`crates/gascand/src/service.rs:3012`). Nothing today would notice. The Arca protocol handoff +names this hazard directly: *"`ListResources` returns unlabelled resources too. Gas Can's drift +detection depends on seeing foreign ones, so filtering engine-side would break it silently."* + +Closing it needs **one** assertion per backend — create a resource out-of-band, assert +`list_resources` reports it — plus a teardown story, since a leaked foreign container on a +developer's machine is a real mess. **That is a separate task, not this one**, and it is written +here so it is not lost. + +## 5. Out of scope + +- **The product-level `gascan-e2e` suite on arca.** P5's *second* exit clause. Apple has 24 + `#[ignore]`d product tests (`apple_apply` 17, `apple_security` 5, `apple_lifecycle` 1, + `apple_recovery` 1); arca has 6. Closing that gap means parameterising the harness — the two + `command()` builders in `apple_common/mod.rs:961` and `arca_common/mod.rs:303` differ by exactly + four `.env()` calls — and it is a larger, separate piece of work. +- **P5.4 / U5**, image digests reaching the engine. Confirmed unresolved at + `docs/status/arca-integration-handoff.md:2242`. +- **Offline / `CERTIFIED_ENGINE_REVISION`**, still `None` at `crates/gascan-arca/src/translate.rs:329`. + Arca-side engine work; gates P7, not P5.3. +- **Production changes to `gascan-arca`.** If arca fails a promoted assertion, that is a finding + to fix as its own work. A conformance suite that is edited until it passes measures nothing. + +## 6. Testing, and the failure mode that would make this lie + +**A conformance suite that silently does not run is worse than none**, because it reports success. +Two mechanisms already exist and both are used rather than reinvented. + +`scripts/ci-check-ignored-tests.sh` diffs the entire `#[ignore]` set against +`tests/ci/expected-ignored-tests.txt` and **fails in both directions** — so a test that vanishes is +caught, not just one that appears. `backend_contract::backend_contract` is already an entry in that +file. Every change here moves that set: apple's entry changes shape and arca gains one. **Updating +the baseline is part of the work, and the guard is what proves the suite is still wired in.** + +Absence of a live prerequisite must **panic, never skip**. +`crates/gascan-arca/tests/live/common/mod.rs:137-140` already rules this, above `required_path` +(`:147`), and states the reason: *"a live test that silently skips is a live test nobody notices +has stopped running."* The same sentence is repeated at `:333-335` on `LiveEngine::start`. The arca +instantiation uses both and inherits the rule. + +**Where each instantiation actually runs, which is asymmetric and worth knowing before relying on +it:** the fake instantiation runs in `cargo test --workspace`, so CI covers it every push. The +**arca** instantiation lands in the live tier CI already executes — +`cargo test -p gascan-arca --test live --no-fail-fast -- --ignored`, `.github/workflows/ci.yml:178` +— so it inherits CI coverage free. The **apple** instantiation runs **nowhere in CI**; no workflow +step passes `--ignored` for `gascan-apple` or `gascan-e2e`. It is a local, manual tier. This design +does not change that, and no claim that "apple passes conformance" should be made without saying +which machine it was run on and when. + +**Proving the extraction is behaviour-preserving.** The fake instantiation must pass before and +after the move with **no edit to any assertion body**. If an assertion had to change, the move was +not a move. + +**Proving a promoted assertion is real.** For each one promoted, revert the behaviour it names in +the fake and confirm the suite fails. An assertion that cannot be made to fail is not testing +anything — this repository has measured that exact outcome, at +`docs/status/START-HERE.md:604`: a guard whose deletion *and* inversion (so that it unlinked +precisely a stranger's file) both left `cargo test -p gascan --lib` green. + +**The hazard that does NOT apply here, recorded so nobody re-imports it.** The e2e harness selects +its backend from the environment and `backend_selection` returns `Apple` when nothing is requested +(`crates/gascan-core/src/backend.rs:167`), so a dropped variable there silently tests the wrong +backend. **Conformance is not exposed to this**: each instantiation constructs its backend +explicitly in code — `AppleBackend::new(ProcessRunner)`, and arca's from a live transport. There is +no selection to get wrong. This matters only if the product-e2e work in §5 is taken up later, where +it applies fully. + +## 7. Acceptance + +1. `gascan-conformance` exists, is a dev-dependency only, and is compiled into no shipped artifact. +2. `gascan-core/src/lib.rs`'s `deny` attribute is unchanged, and no `#[allow(clippy::unwrap_used)]` + was added anywhere to accommodate this work. +3. The 65-line duplicate at `gascan-apple/tests/live/backend_contract.rs` is **deleted**, replaced + by a call into the shared suite. +4. Three instantiations exist: fake, apple, arca. +5. `tests/ci/expected-ignored-tests.txt` is updated and `scripts/ci-check-ignored-tests.sh` passes. +6. Every promoted assertion has been shown to fail when the behaviour it names is reverted in the + fake. +7. `cargo fmt --all --check`, `cargo clippy --workspace --all-targets -- -D warnings`, and + `cargo test --workspace` are clean. +8. **Arca's result against the suite is recorded as a finding, not as a pass criterion.** If arca + fails a promoted assertion, this task's deliverable is the *measurement* and its write-up. The + fix is separate work, and forcing green here by weakening an assertion is the one outcome that + makes the whole exercise worthless. From 4a935fd86bbd920bef0e43f789ca9cf13c4b970e Mon Sep 17 00:00:00 2001 From: Richard Kiene Date: Thu, 20 Aug 2026 11:20:00 -0700 Subject: [PATCH 2/3] docs: P5.3's implementation plan, in eight tasks Implements `docs/superpowers/specs/2026-08-20-backend-conformance-suite-design.md`. Eight tasks: create the crate, move the contract in and instantiate against the fake, strip the duplicate out of gascan-core, instantiate against apple and arca, promote start/stop idempotence, triage the remaining candidates, record the measurement. **Filling the plan's own holes is what found the blockers.** A first draft left three `/* ... */` markers in the arca task for the harness's constructor names, annotated as "read lifecycle.rs". Reading it surfaced three constraints that would each have failed the first run: - The manifest needs `user = 'root'`. The base layout is a stock alpine with no `workspace` account, and the engine hands `UserMode::Workspace` to `createContainer` as the literal string `workspace`, so `start` would fail on the image rather than on anything under test (`lifecycle.rs:24-30`). - The image needs a rewritten `Cmd`. Alpine's own is `/bin/sh`, which exits immediately with no tty; the contract does start -> exec -> stop and needs a container still running (`lifecycle.rs:35-37`). `layout_running` (`gascan-oci-fixture/src/lib.rs:39`) rewrites it without touching the rootfs. - The network must be `networked`, not the fixture's default `offline`. Offline is the capability the pinned engine does not honour, so an offline request would test the refuted property by accident. That third fact changed the crate's API: `for_image` takes a whole manifest rather than a network string, matching `policy_request_from_manifest(name, image, manifest)` at `live/common/mod.rs:718` -- `user` is a manifest fact and a network-only parameter cannot carry it. The tier already documents the reasoning: "The manifest is the only knob, deliberately." One deliberate change from the fixtures being moved: `RuntimeVersion::new(1, 0, 0)` becomes `(1, 1, 0)`. Verified safe -- `capabilities.version` reaches only an error message (`policy.rs:422`, inside `PolicyError::OfflineUnsupported`) and neither backend validates a request's version. 1.1.0 matches apple's floor (`probe.rs:36`) and the value arca's tier already uses. Task 5 step 6 has no expected result written for it, on purpose. Arca's run is the measurement the plan exists for, and step 7 requires committing the failing test with the failure quoted rather than weakening the contract. --- .../2026-08-20-backend-conformance-suite.md | 724 ++++++++++++++++++ 1 file changed, 724 insertions(+) create mode 100644 docs/superpowers/plans/2026-08-20-backend-conformance-suite.md diff --git a/docs/superpowers/plans/2026-08-20-backend-conformance-suite.md b/docs/superpowers/plans/2026-08-20-backend-conformance-suite.md new file mode 100644 index 0000000..e1c99a8 --- /dev/null +++ b/docs/superpowers/plans/2026-08-20-backend-conformance-suite.md @@ -0,0 +1,724 @@ +# Backend Conformance Suite Implementation Plan + +> **For agentic workers:** REQUIRED SUB-SKILL: Use superpowers:subagent-driven-development (recommended) or superpowers:executing-plans to implement this plan task-by-task. Steps use checkbox (`- [ ]`) syntax for tracking. + +**Goal:** Make the existing `RuntimeBackend` conformance contract importable from a shared crate, and run it against all three backends — fake, apple, and arca — so P5's first exit clause can be measured rather than assumed. + +**Architecture:** A new dev-dependency-only crate, `gascan-conformance`, owns the contract walk and the `CreateRequest` fixtures. Each backend instantiates it from its own test target. `gascan-core/tests/backend_contract.rs` keeps only what genuinely tests the double's controllability. The contract takes a *fixture* rather than building one, because apple and arca pin different images. + +**Tech Stack:** Rust 2024 (workspace `resolver = "3"`), `tokio` test harness, `async_trait`, `tempfile`, `camino`. + +**Spec:** `docs/superpowers/specs/2026-08-20-backend-conformance-suite-design.md` — read it first; this plan argues from it. + +## Global Constraints + +- **Never weaken `gascan-core`'s lint gate.** `crates/gascan-core/src/lib.rs:2` is `#![deny(clippy::expect_used, clippy::panic, clippy::unwrap_used)]`. Do not add `#[allow]` for any of these anywhere, and do not edit that line. The whole reason `gascan-conformance` exists is to avoid it. +- **`PolicyCompiler` is the only way to build a `CreateRequest`.** Its fields are `pub(crate)` to `gascan-core` and it derives no `Deserialize`. Do not add a constructor, do not widen visibility. +- **A missing live prerequisite panics; it never skips.** Rule and rationale at `crates/gascan-arca/tests/live/common/mod.rs:137-140`. +- **Never edit an assertion to make a backend pass.** If arca fails, that is the deliverable. See Task 8. +- CI runs `cargo fmt --all --check`, `cargo clippy --workspace --all-targets -- -D warnings`, and `cargo test --workspace` (`.github/workflows/ci.yml:51`, `:54`, `:57`). All three must be clean at every commit. +- Workspace lints are only `unsafe_code = "forbid"` (`Cargo.toml`, `[workspace.lints.rust]`). The new crate uses `[lints] workspace = true` like its siblings. + +--- + +## File Structure + +| File | Responsibility | +|---|---| +| `crates/gascan-conformance/Cargo.toml` | New crate manifest; `gascan-core` as a normal dependency | +| `crates/gascan-conformance/src/lib.rs` | `backend_contract()`, `CreateRequestFixture`, `capabilities()` | +| `crates/gascan-conformance/tests/fake.rs` | Instantiation 1 — `FakeRuntime` | +| `crates/gascan-apple/tests/live/backend_contract.rs` | Instantiation 2 — replaces a 65-line hand-rolled duplicate | +| `crates/gascan-arca/tests/live/conformance.rs` | Instantiation 3 — new | +| `crates/gascan-arca/tests/live.rs` | Add `mod conformance;` | +| `crates/gascan-core/tests/backend_contract.rs` | Loses the generic fn and promoted tests; keeps fake-only ones | +| `crates/gascan-core/tests/common/mod.rs` | Loses the fixtures that move; keeps whatever other tests still need | +| `tests/ci/expected-ignored-tests.txt` | The `#[ignore]` baseline; changes in Tasks 6 and 7 | +| `Cargo.toml` | Add the new crate to `members` | + +--- + +### Task 1: Create the `gascan-conformance` crate with its fixtures + +**Files:** +- Create: `crates/gascan-conformance/Cargo.toml` +- Create: `crates/gascan-conformance/src/lib.rs` +- Modify: `Cargo.toml` (workspace `members`) + +**Interfaces:** +- Consumes: `gascan_core::{manifest::Manifest, policy::PolicyCompiler, runtime::*, sandbox::SandboxSpec}` +- Produces: `gascan_conformance::{capabilities, CreateRequestFixture}`. `CreateRequestFixture::pinned(name: &str, network: &str) -> Self`, `CreateRequestFixture::for_image(name: &str, image: &str, manifest: &str) -> Self`, `fixture.request() -> CreateRequest`, and `Deref`. + +**`for_image` takes a whole manifest, not a network string, and the argument order matches `policy_request_from_manifest(name, image, manifest)` at `crates/gascan-arca/tests/live/common/mod.rs:718`.** That harness documents why: *"The manifest is the only knob, deliberately. Ports and the guest user are manifest facts and nothing else in this tier may set them."* Arca needs `user = 'root'` in its manifest (Task 5), which a network-only parameter cannot express. + +**Why two constructors:** `PolicyCompiler::compile` pins the approved workspace image (`policy.rs:85-90`), which no engine under test holds. Arca's live tier seeds a store with `arca-engine image load` and must ask for what it seeded, so it needs `compile_for_image` (`policy.rs:92-98`). This is recorded at `crates/gascan-arca/tests/live/common/mod.rs:691-694`. + +- [ ] **Step 1: Add the crate to the workspace** + +In `Cargo.toml`, add `"crates/gascan-conformance"` to `members`, keeping the list alphabetical: + +```toml +members = ["crates/gascan", "crates/gascan-apple", "crates/gascan-arca", "crates/gascan-conformance", "crates/gascan-core", "crates/gascan-e2e", "crates/gascan-engine-proto", "crates/gascan-inherited-fd", "crates/gascan-oci-fixture", "crates/gascan-proto", "crates/gascand"] +``` + +- [ ] **Step 2: Write the manifest** + +Create `crates/gascan-conformance/Cargo.toml`: + +```toml +[package] +name = "gascan-conformance" +version.workspace = true +edition.workspace = true +license.workspace = true +rust-version.workspace = true +publish = false + +[dependencies] +camino.workspace = true +gascan-core = { path = "../gascan-core" } +tempfile = "3" + +[dev-dependencies] +tokio = { workspace = true, features = ["macros", "rt", "rt-multi-thread", "sync", "time"] } + +[lints] +workspace = true +``` + +If `version.workspace = true` fails because the workspace defines no shared version, copy the literal version from `crates/gascan-core/Cargo.toml` instead. Check with `sed -n '1,10p' crates/gascan-core/Cargo.toml`. + +- [ ] **Step 3: Write the fixtures** + +Create `crates/gascan-conformance/src/lib.rs`. This is `crates/gascan-core/tests/common/mod.rs` with a second constructor added — `capabilities()` and the compile body are copied verbatim so behaviour cannot drift: + +```rust +//! Backend conformance: one contract, run against every `RuntimeBackend`. +//! +//! This crate exists because `gascan-core/src/lib.rs:2` denies +//! `clippy::unwrap_used`, and a conformance suite is built from unwrapping +//! assertions. It is a dev-dependency of its consumers and ships nowhere. + +use camino::Utf8Path; +use gascan_core::manifest::Manifest; +use gascan_core::policy::PolicyCompiler; +use gascan_core::runtime::{CreateRequest, NetworkIsolation, RuntimeCapabilities, RuntimeVersion}; +use gascan_core::sandbox::SandboxSpec; +use std::ops::Deref; + +pub struct CreateRequestFixture { + _root: tempfile::TempDir, + request: CreateRequest, +} + +impl CreateRequestFixture { + /// A request against the approved workspace image. + /// + /// Correct for the fake and for apple. **Wrong for a live engine**, whose + /// store holds only what the tier seeded -- use [`Self::for_image`] there. + pub fn pinned(name: &str, network: &str) -> Self { + assert!(matches!(network, "offline" | "networked")); + Self::build(name, &format!("version = 1\nnetwork = '{network}'\n"), None) + } + + /// A request against `image`, for a backend whose store was seeded with it. + /// + /// The manifest is the only knob, matching `policy_request_from_manifest` + /// in arca's live tier: the guest user and any ports are manifest facts, + /// and a caller reaching around them would build a request gascan itself + /// cannot produce. + pub fn for_image(name: &str, image: &str, manifest: &str) -> Self { + Self::build(name, manifest, Some(image)) + } + + pub fn request(&self) -> CreateRequest { + self.request.clone() + } + + fn build(name: &str, manifest_text: &str, image: Option<&str>) -> Self { + let temp = tempfile::tempdir().expect("temporary backend-contract root"); + let root = Utf8Path::from_path(temp.path()).expect("UTF-8 temporary path"); + std::fs::write(root.join("gascan.toml"), manifest_text) + .expect("write backend-contract manifest"); + let manifest = Manifest::load(root).expect("load backend-contract manifest"); + let spec = SandboxSpec::from_root(name, root, manifest).expect("build sealed sandbox spec"); + let request = match image { + None => PolicyCompiler::compile(spec, &capabilities()), + Some(image) => PolicyCompiler::compile_for_image(spec, &capabilities(), image), + } + .expect("compile backend-contract policy"); + Self { + _root: temp, + request, + } + } +} + +impl Deref for CreateRequestFixture { + type Target = CreateRequest; + + fn deref(&self) -> &Self::Target { + &self.request + } +} + +/// Every flag true. The compiler gates on what a runtime CLAIMS, and the +/// contract only needs a well-formed request; what is under test is the +/// backend's behaviour, not the compiler's gating. +pub fn capabilities() -> RuntimeCapabilities { + RuntimeCapabilities { + version: RuntimeVersion::new(1, 1, 0), + bind_mounts: true, + named_volumes: true, + tty: true, + signals: true, + loopback_publish: true, + resource_limits: true, + offline: NetworkIsolation::Proven, + } +} +``` + +**Note the one deliberate change from the original:** `RuntimeVersion::new(1, 0, 0)` becomes `(1, 1, 0)`. Verified safe — `capabilities.version` reaches only an error message (`crates/gascan-core/src/policy.rs:422`, inside `PolicyError::OfflineUnsupported`), and neither backend validates a request's version. `1.1.0` matches apple's floor at `crates/gascan-apple/src/probe.rs:36` and the value arca's live tier already uses. + +- [ ] **Step 4: Verify it builds and lints** + +Run: `cargo build -p gascan-conformance && cargo clippy -p gascan-conformance --all-targets -- -D warnings` +Expected: both exit 0, no warnings. + +- [ ] **Step 5: Commit** + +```bash +git add Cargo.toml crates/gascan-conformance/ +git commit -m "feat: a conformance crate, because gascan-core denies the unwraps a suite needs" +``` + +--- + +### Task 2: Move the contract in and instantiate it against the fake + +**Files:** +- Modify: `crates/gascan-conformance/src/lib.rs` +- Create: `crates/gascan-conformance/tests/fake.rs` + +**Interfaces:** +- Consumes: `CreateRequestFixture` from Task 1. +- Produces: `pub async fn backend_contract(backend: &dyn RuntimeBackend, fixture: &CreateRequestFixture)`. + +**The signature change from the original:** the existing `backend_contract` at `crates/gascan-core/tests/backend_contract.rs:149` builds its own request via `create_request("contract")`. It must take one instead, because apple and arca pin different images (Task 1's note). + +- [ ] **Step 1: Write the failing test** + +Create `crates/gascan-conformance/tests/fake.rs`: + +```rust +use gascan_conformance::{CreateRequestFixture, backend_contract, capabilities}; +use gascan_core::fake_runtime::FakeRuntime; +use gascan_core::runtime::RuntimeBackend; + +#[tokio::test] +async fn fake_runtime_satisfies_the_backend_contract() { + let backend: Box = Box::new(FakeRuntime::new(capabilities())); + let fixture = CreateRequestFixture::pinned("contract", "offline"); + backend_contract(backend.as_ref(), &fixture).await; +} +``` + +- [ ] **Step 2: Run it to verify it fails** + +Run: `cargo test -p gascan-conformance --test fake` +Expected: FAIL to compile — `cannot find function 'backend_contract' in crate 'gascan_conformance'`. + +- [ ] **Step 3: Move the contract into the crate** + +Append to `crates/gascan-conformance/src/lib.rs`. The body is copied **verbatim** from `crates/gascan-core/tests/backend_contract.rs:149-179`; only the signature and the first two lines change, so that a diff shows the move rather than a rewrite: + +```rust +use gascan_core::runtime::{ + ContainerState, ExecInput, ExecOutput, ExecRequest, RemoveRequest, ResourceKind, + RuntimeBackend, +}; + +/// The contract every `RuntimeBackend` owes, whatever it is implemented over. +/// +/// `fixture` is a parameter and not built here because `PolicyCompiler::compile` +/// pins the approved workspace image, which a live engine's seeded store does +/// not hold -- see `CreateRequestFixture::for_image`. +pub async fn backend_contract(backend: &dyn RuntimeBackend, fixture: &CreateRequestFixture) { + let id = fixture.id().clone(); + assert_eq!(backend.inspect(&id).await.unwrap(), None); + let created = backend.create(fixture.request()).await.unwrap(); + assert!( + created + .created() + .iter() + .any(|resource| resource.kind() == ResourceKind::Container) + ); + assert_eq!( + backend.inspect(&id).await.unwrap().unwrap().state, + ContainerState::Stopped + ); + backend.start(&id).await.unwrap(); + let mut session = backend + .exec(ExecRequest::fixture(id.clone(), ["true"])) + .await + .unwrap(); + session.send(ExecInput::Close).await.unwrap(); + assert_eq!( + session.next().await.unwrap().unwrap(), + ExecOutput::Exit { code: 0, signal: 0 } + ); + backend.stop(&id).await.unwrap(); + backend + .remove(RemoveRequest::from_resources(created.created().to_vec()).unwrap()) + .await + .unwrap(); + assert_eq!(backend.inspect(&id).await.unwrap(), None); +} +``` + +If `fixture.id()` does not resolve, it is reached through the `Deref` to `CreateRequest`; confirm with `grep -n "pub fn id" crates/gascan-core/src/runtime.rs`. + +- [ ] **Step 4: Run the test to verify it passes** + +Run: `cargo test -p gascan-conformance --test fake` +Expected: PASS, `1 passed`. + +- [ ] **Step 5: Prove the contract can fail** + +This guards against a contract that asserts nothing. Temporarily edit the assertion after `stop`: + +```rust + assert_eq!(backend.inspect(&id).await.unwrap(), None); +``` + +to compare against `Some(...)`-shaped nonsense — simplest is to change the final line to `assert!(backend.inspect(&id).await.unwrap().is_some());` + +Run: `cargo test -p gascan-conformance --test fake` +Expected: FAIL. Then **revert the edit** and re-run to confirm PASS. + +- [ ] **Step 6: Commit** + +```bash +git add crates/gascan-conformance/ +git commit -m "feat: the backend contract moves to where every backend can reach it" +``` + +--- + +### Task 3: Strip the moved code out of `gascan-core` + +**Files:** +- Modify: `crates/gascan-core/tests/backend_contract.rs` (delete `:149-179` and the fake trait-object test) +- Modify: `crates/gascan-core/tests/common/mod.rs` + +**Interfaces:** +- Consumes: nothing new. +- Produces: nothing. This task only removes duplication. + +- [ ] **Step 1: Delete the generic function and its fake instantiation** + +Remove `pub async fn backend_contract(...)` (`:149-179`) and `fake_runtime_satisfies_backend_contract_through_trait_object` (which calls it — locate with `grep -n "fake_runtime_satisfies_backend_contract_through_trait_object" crates/gascan-core/tests/backend_contract.rs`). Task 2's `tests/fake.rs` replaces both. + +- [ ] **Step 2: Remove now-unused fixtures and imports** + +`create_request`, `create_request_with_network`, `capabilities` and `CreateRequestFixture` may still be used by the fake-only tests that remain in this file, and by other test files in `crates/gascan-core/tests/`. **Check before deleting:** + +Run: `grep -rn "create_request\|capabilities()\|CreateRequestFixture" crates/gascan-core/tests/` + +Delete from `common/mod.rs` only what nothing references. If everything still references them, `common/mod.rs` is unchanged and that is a correct outcome — the duplication with `gascan-conformance` is then deliberate and short-lived, and Task 9 revisits it. + +- [ ] **Step 3: Verify the crate still tests clean** + +Run: `cargo test -p gascan-core` +Expected: PASS. The count drops by exactly 1 (the removed trait-object test). Record the before and after numbers in the commit message. + +- [ ] **Step 4: Verify no warnings** + +Run: `cargo clippy -p gascan-core --all-targets -- -D warnings` +Expected: exit 0. Unused-import warnings here are the signal that Step 2 missed something. + +- [ ] **Step 5: Commit** + +```bash +git add crates/gascan-core/ +git commit -m "refactor: gascan-core stops owning a contract every backend needs" +``` + +--- + +### Task 4: Instantiate against apple, deleting the hand-rolled duplicate + +**Files:** +- Modify: `crates/gascan-apple/tests/live/backend_contract.rs` (replace all 65 lines) +- Modify: `crates/gascan-apple/Cargo.toml` (add the dev-dependency) + +**Interfaces:** +- Consumes: `gascan_conformance::{backend_contract, CreateRequestFixture}`. +- Produces: nothing later tasks use. + +- [ ] **Step 1: Add the dev-dependency** + +In `crates/gascan-apple/Cargo.toml`, under `[dev-dependencies]`: + +```toml +gascan-conformance = { path = "../gascan-conformance" } +``` + +- [ ] **Step 2: Replace the file** + +`crates/gascan-apple/tests/live/backend_contract.rs` becomes: + +```rust +use gascan_apple::{AppleBackend, ProcessRunner}; +use gascan_conformance::{CreateRequestFixture, backend_contract}; +use std::time::{SystemTime, UNIX_EPOCH}; + +#[tokio::test] +#[ignore = "requires Apple silicon macOS 26+ with container service and locked workspace image"] +async fn backend_contract_holds_on_apple() { + let nonce = SystemTime::now() + .duration_since(UNIX_EPOCH) + .unwrap() + .as_nanos(); + let name = format!("gascan-live-backend-{}-{nonce}", std::process::id()); + let fixture = CreateRequestFixture::pinned(&name, "offline"); + backend_contract(&AppleBackend::new(ProcessRunner), &fixture).await; +} +``` + +**Two behaviours the old file had that the shared contract also has**, so nothing is lost: the `inspect`-absent bookends and the create/start/stop/remove walk. **One it had that the contract does not** — a final `list_resources()` check that no resource name starts with the sandbox name. Add it after the `backend_contract` call rather than dropping it: + +```rust + let backend = AppleBackend::new(ProcessRunner); + backend_contract(&backend, &fixture).await; + assert!( + !backend + .list_resources() + .await + .unwrap() + .iter() + .any(|resource| resource.name().starts_with(&name)) + ); +``` + +with `use gascan_core::runtime::RuntimeBackend;` added for `list_resources`. Restructure the test to bind `backend` once, as above. + +**The old file called `start` twice and `stop` twice** to assert idempotence. The shared contract calls each once. Do **not** silently drop that — it is a real assertion and Task 7 promotes it. Until then, note it in the commit message as temporarily uncovered on apple. + +- [ ] **Step 3: Verify it compiles** + +Run: `cargo test -p gascan-apple --no-run` +Expected: compiles clean. It cannot be *run* without an Apple runtime; that is Step 5. + +- [ ] **Step 4: Update the ignored-test baseline** + +The test's name changed from `backend_contract` to `backend_contract_holds_on_apple`, so the baseline must change or CI fails. + +Run: `./scripts/ci-check-ignored-tests.sh` +Expected: FAIL, naming both the removed and added entries. + +Edit `tests/ci/expected-ignored-tests.txt`: replace the `backend_contract::backend_contract` line with `backend_contract::backend_contract_holds_on_apple`, keeping the file's existing sort order. + +Run: `./scripts/ci-check-ignored-tests.sh` +Expected: PASS. + +- [ ] **Step 5: Run it for real, if this machine can** + +Prerequisites, verified present on the maintainer's machine on 2026-08-20: `container` CLI 1.1.0 at `/usr/local/bin/container` with `container system status` reporting `running`, and two `ghcr.io/liquescent-development/gascan/workspace` images. + +Run: `cargo test -p gascan-apple --test live -- --ignored backend_contract_holds_on_apple` +Expected: PASS. + +**If it fails, stop and read the failure before changing anything.** A failure here means either the extraction changed behaviour (Task 2's fault) or apple never satisfied the shared contract's extra assertions (a finding). Record which, in the commit. + +**If this machine cannot run it, say so explicitly in the commit message** — do not imply it passed. Apple's tier runs in no CI job (no workflow step passes `--ignored` for `gascan-apple`), so this run is the only evidence that will ever exist. + +- [ ] **Step 6: Commit** + +```bash +git add crates/gascan-apple/ tests/ci/expected-ignored-tests.txt +git commit -m "refactor: apple runs the shared contract instead of its own copy of it" +``` + +--- + +### Task 5: Instantiate against arca + +**Files:** +- Create: `crates/gascan-arca/tests/live/conformance.rs` +- Modify: `crates/gascan-arca/tests/live.rs` (add `mod conformance;`) +- Modify: `crates/gascan-arca/Cargo.toml` (add the dev-dependency) + +**Interfaces:** +- Consumes: `gascan_conformance::{backend_contract, CreateRequestFixture}`, and the live tier's existing engine harness in `crates/gascan-arca/tests/live/common/mod.rs`. +- Produces: nothing later tasks use. + +**Read first:** `crates/gascan-arca/tests/live/lifecycle.rs` — it is the closest existing test and shows how the tier starts an engine, seeds an image, and builds a backend. Copy its setup shape rather than inventing one. + +- [ ] **Step 1: Add the dev-dependency** + +In `crates/gascan-arca/Cargo.toml`, under `[dev-dependencies]`: + +```toml +gascan-conformance = { path = "../gascan-conformance" } +``` + +- [ ] **Step 2: Write the test** + +Create `crates/gascan-arca/tests/live/conformance.rs`. **Three things differ from apple and each one would fail the run if missed:** + +1. **`network = 'networked'`, not `'offline'`.** `lifecycle.rs:31` uses `networked`, and offline is exactly the capability the pinned engine does not honour (`docs/evidence/2026-08-18-arca-engine-offline.md`). An offline request here would test the refuted property by accident. +2. **`user = 'root'`.** `lifecycle.rs:24-30` records why: the base layout is a stock alpine with no `workspace` account, and the engine translates `UserMode::Workspace` to the literal string `workspace` and hands it to `createContainer` — so `start` would fail on the image rather than on anything under test. +3. **A staying-up image, not the bare base.** Alpine's own `Cmd` is `/bin/sh`, which exits immediately with no tty attached (`lifecycle.rs:35-37`). The contract does `start` → `exec` → `stop`, so the container has to still be there. `layout_running` rewrites the `Cmd` without rebuilding the rootfs. + +```rust +use crate::common::{LiveEngine, base_oci_layout, layout_running}; +use camino::Utf8Path; +use gascan_arca::ArcaBackend; +use gascan_conformance::{CreateRequestFixture, backend_contract}; + +/// The tag the derived layout is loaded under. +const TAG: &str = "gascan-conformance:latest"; + +/// `user = 'root'` because the base layout is a stock alpine with no +/// `workspace` account -- see `lifecycle.rs`'s note on the same constant. +const MANIFEST: &str = "version = 1\nnetwork = 'networked'\nuser = 'root'\n"; + +#[tokio::test] +#[ignore = "requires a built arca-engine, a kernel, a vminit layout and a base OCI layout"] +async fn backend_contract_holds_on_arca() { + let temp = tempfile::tempdir().expect("a temporary layout root"); + let destination = Utf8Path::from_path(temp.path()).expect("a utf-8 temporary path"); + let layout = layout_running( + &base_oci_layout(), + destination, + TAG, + &["sh", "-c", "while :; do sleep 1; done"], + ); + let engine = LiveEngine::start_with_images(&[layout.as_path()]).await; + let backend = ArcaBackend::new(engine.transport().await); + let fixture = CreateRequestFixture::for_image("conformance", TAG, MANIFEST); + backend_contract(&backend, &fixture).await; +} +``` + +Signatures this uses, all re-derived on 2026-08-20: `layout_running(base: &Utf8Path, destination: &Utf8Path, tag: &str, command: &[&str]) -> Utf8PathBuf` (`crates/gascan-oci-fixture/src/lib.rs:39`, re-exported through `crate::common`); `LiveEngine::start_with_images(layouts: &[&Utf8Path])` (`live/common/mod.rs:347`); `LiveEngine::transport(&self) -> ChannelTransport` (`:447`); `base_oci_layout() -> Utf8PathBuf` (`:156`); `ArcaBackend::new(engine.transport().await)` (`lifecycle.rs:14-16`). + +- [ ] **Step 3: Register the module** + +Add `mod conformance;` to `crates/gascan-arca/tests/live.rs`, in the file's existing alphabetical position. + +- [ ] **Step 4: Verify it compiles** + +Run: `cargo test -p gascan-arca --test live --no-run` +Expected: compiles clean. + +- [ ] **Step 5: Update the ignored-test baseline** + +Run: `./scripts/ci-check-ignored-tests.sh` +Expected: FAIL, naming the added entry. + +Add `conformance::backend_contract_holds_on_arca` to `tests/ci/expected-ignored-tests.txt` in sort order, then re-run. +Expected: PASS. + +- [ ] **Step 6: Run it — this is the measurement the whole plan exists for** + +Set the four variables the tier requires (`crates/gascan-arca/tests/live/common/mod.rs` names each and panics with a directive when absent), then: + +Run: `cargo test -p gascan-arca --test live -- --ignored backend_contract_holds_on_arca` + +**There is no expected result.** Whatever happens is the finding. Record the exact output. + +- [ ] **Step 7: Commit the result, whichever it is** + +If it passes, say so with the command and the engine revision from `engine/arca-pin.json`. + +If it fails, **commit the failing test anyway**, with the failure quoted in full in the commit message and the assertion that failed named. Do not weaken the contract, do not add `#[ignore]` beyond the tier's existing one, and do not "fix" arca in this commit — that is separate work, and conflating them destroys the evidence. + +```bash +git add crates/gascan-arca/ tests/ci/expected-ignored-tests.txt +git commit -m "test: arca meets the shared backend contract, and here is what it did" +``` + +--- + +### Task 6: Promote start/stop idempotence + +**Files:** +- Modify: `crates/gascan-conformance/src/lib.rs` +- Modify: `crates/gascan-core/tests/backend_contract.rs` (remove the promoted half) + +**Interfaces:** +- Consumes: `backend_contract` from Task 2. +- Produces: no new public names; `backend_contract`'s body grows. + +**Why this one first:** apple's deleted file already asserted it (Task 4 recorded it as temporarily uncovered), so promoting it closes a regression this plan itself opened. It is also the clearest case of the §4 rule — idempotent `start`/`stop` is implemented separately by every backend. + +- [ ] **Step 1: Read the source test** + +Run: `grep -n "duplicate_create_is_rejected_and_start_stop_are_idempotent" -A 40 crates/gascan-core/tests/backend_contract.rs` + +Separate what any backend owes (repeated `start` and `stop` succeed; a second `create` of the same id is rejected) from what the fake happens to do (any assertion reaching `calls()`, `outcomes()`, or a `seed_*` method — those cannot move). + +- [ ] **Step 2: Add the generic half to the contract** + +In `backend_contract`, replace the single `start`/`stop` calls with doubled ones: + +```rust + backend.start(&id).await.unwrap(); + backend.start(&id).await.unwrap(); +``` + +and + +```rust + backend.stop(&id).await.unwrap(); + backend.stop(&id).await.unwrap(); +``` + +- [ ] **Step 3: Verify the fake still passes** + +Run: `cargo test -p gascan-conformance --test fake` +Expected: PASS. + +- [ ] **Step 4: Prove the new assertion can fail** + +In `crates/gascan-core/src/fake_runtime.rs`, make `start` fail when the sandbox is already running — find it with `grep -n "async fn start" crates/gascan-core/src/fake_runtime.rs` and return `RuntimeError::InvalidState { .. }` on the second call. + +Run: `cargo test -p gascan-conformance --test fake` +Expected: **FAIL** on the doubled `start`. + +**Then revert the fake edit** and re-run. +Expected: PASS. + +A promoted assertion that cannot be made to fail is testing nothing. This repository has measured exactly that outcome — `docs/status/START-HERE.md:604` records a guard whose deletion *and* inversion both left the suite green. + +- [ ] **Step 5: Remove the now-duplicated half from `gascan-core`** + +Delete only the start/stop-idempotence assertions from `duplicate_create_is_rejected_and_start_stop_are_idempotent`. **Keep the duplicate-create rejection there** unless Step 1 showed it needs no fake-only machinery — if it needs none, promote it too, repeating Steps 2-4 for it. + +Run: `cargo test -p gascan-core` +Expected: PASS. + +- [ ] **Step 6: Commit** + +```bash +git add crates/gascan-conformance/ crates/gascan-core/ +git commit -m "test: start and stop are idempotent on every backend, not just the fake" +``` + +--- + +### Task 7: Triage the remaining promotion candidates + +**Files:** +- Modify: `crates/gascan-conformance/src/lib.rs` +- Modify: `crates/gascan-core/tests/backend_contract.rs` +- Modify: `docs/superpowers/specs/2026-08-20-backend-conformance-suite-design.md` (record the outcome) + +**Interfaces:** +- Consumes: `backend_contract` from Task 2. +- Produces: no new public names. + +**This task is a judgment, and the plan supplies the criterion rather than the answer.** The spec's §4 rule: *an assertion earns promotion only if the behaviour it names is implemented separately by each backend.* Shared code tested once is done — that is why the ownership assertions are **not** on this list; `classify_resource_ownership` is one pure function at `crates/gascan-core/src/runtime.rs:85-103`, already tested exhaustively by `crates/gascan-core/tests/resource_ownership.rs`. + +**Candidates, in the order to take them:** + +| Test | The question to answer | +|---|---| +| `exec_and_logs_preserve_binary_bytes_and_exact_exit_code` | Does it need `set_exec_result`/`queue_exec_results`? Those are fake-only. Can the same property be asserted by exec'ing a real command that emits known bytes? | +| `exec_session_is_live_bidirectional_and_emits_one_exit` | Same question; bidirectional streaming against a real backend needs a real interactive command. | +| `create_collision_reports_resources_created_before_the_collision` | Needs a pre-existing colliding resource. On a real backend, can that be produced by creating twice? | +| `offline_fake_create_has_no_managed_network` | Real for apple. **Arca cannot honour offline** — if promoted, arca's instantiation must keep `networked` and this assertion must be conditional, which is a design change. Consider leaving it fake-and-apple-only. | +| `networked_fake_create_reports_network_then_volumes_then_container` | The *ordering* is asserted through the fake's call recorder. Only the *effect* is portable. Likely stays. | +| `persistent_logs_are_isolated_by_exact_sandbox_id` | Does isolation-by-id need `FakeRuntime::persistent`? If so it is fake-only. | + +- [ ] **Step 1: Take each candidate in turn** + +For each row: read the test, apply the criterion, and reach one of two outcomes. + +- [ ] **Step 2: If it promotes — repeat Task 6's cycle exactly** + +Add the generic assertion to `backend_contract`; run the fake instantiation and see it pass; **break the behaviour in `fake_runtime.rs` and see it fail**; revert; remove the duplicated half from `gascan-core`; commit one candidate per commit. + +- [ ] **Step 3: If it does not promote — record why, in the spec** + +Append a row to §3's fake-only list in `docs/superpowers/specs/2026-08-20-backend-conformance-suite-design.md` naming the test and the fake-only machinery it depends on. A candidate silently left behind is indistinguishable from one that was forgotten. + +- [ ] **Step 4: Re-run everything** + +Run: `cargo test -p gascan-conformance && cargo test -p gascan-core && ./scripts/ci-check-ignored-tests.sh` +Expected: all PASS. + +- [ ] **Step 5: Reconcile the count** + +The spec estimated **6-8** promotions including Task 6's. Count what actually promoted. **If the real number is outside that range, update the spec's §3 rather than leaving the estimate standing** — a spec whose numbers went stale inside its own implementation is this project's most-repeated failure. + +- [ ] **Step 6: Commit the spec update** + +```bash +git add docs/superpowers/specs/2026-08-20-backend-conformance-suite-design.md +git commit -m "docs: what promoted, what did not, and the machinery that decided each" +``` + +--- + +### Task 8: Record the measurement and close out + +**Files:** +- Create: `docs/evidence/2026-08-20-backend-conformance.md` +- Modify: `docs/status/START-HERE.md` + +**Interfaces:** +- Consumes: Task 5's arca result and Task 7's promotion outcome. +- Produces: the durable record. + +**The spec's acceptance criterion 8 is the one under pressure here:** arca's result is a *finding*, not a pass criterion. If arca failed, the deliverable is the measurement and its write-up — not a green suite. + +- [ ] **Step 1: Write the evidence document** + +`docs/evidence/2026-08-20-backend-conformance.md`, following the shape of `docs/evidence/2026-08-18-arca-engine-offline.md`. It must state: the exact command run for each backend, the engine revision from `engine/arca-pin.json`, the machine, the date, what passed, what failed, and — for anything not run — that it was not run and why. Never write a counterfactual as an event. + +- [ ] **Step 2: Update the handoff** + +In `docs/status/START-HERE.md`, update the queue entry for P5.3 to record it as done or partially done, with the evidence document referenced. If arca failed an assertion, that becomes a new open item naming the assertion and pointing at the evidence. + +- [ ] **Step 3: Full verification, against CI's own step list** + +Run each, and record the exit code of each: + +```bash +cargo fmt --all --check +cargo clippy --workspace --all-targets -- -D warnings +cargo test --workspace +./scripts/ci-check-ignored-tests.sh +``` + +Expected: all exit 0. Note that `cargo test --workspace` on this machine has a measured ~28% failure rate from pre-existing flakes (`START-HERE.md`, THE NINTH MECHANISM) — a failure in `gascan --lib` that is not in the crates this plan touched is very likely one of those, and should be checked against that list before being treated as a regression. + +- [ ] **Step 4: Confirm the crate ships nowhere** + +Run: `cargo tree -p gascan --edges normal | grep gascan-conformance || echo "not in gascan's normal dependency graph"` +Expected: the `echo` branch. Repeat for `gascand`. A conformance crate reachable from a shipped binary is a packaging defect. + +- [ ] **Step 5: Commit and open the PR** + +```bash +git add docs/ +git commit -m "docs: what the conformance suite measured on each backend" +``` + +Open a PR against `main`. **Never merge to `main` directly.** The PR body states, for each of the three backends, whether the contract was run, on what, and with what result. + +--- + +## Self-Review + +**Spec coverage:** §1 (contract exists, needs relocating) → Tasks 2-3. §2 (separate crate, why not `gascan-core`) → Task 1. §3 (promote/fake-only split) → Tasks 6-7. §4 (ownership overbuild rejected) → Task 7's preamble, which restates the criterion and excludes ownership by name. §5 (out of scope) → nothing implements it, correctly; the product-e2e work is absent from every task. §6 (testing, ignored-set guard, the fail-open hazard not applying) → Tasks 4-5 update the baseline, Task 8 Step 3 runs the guard. §7 (acceptance, all 8 criteria) → criterion 1 Task 1, 2 Global Constraints, 3 Task 4, 4 Tasks 2/4/5, 5 Tasks 4/5/8, 6 Tasks 6/7, 7 Task 8 Step 3, 8 Task 5 Step 7 and Task 8. + +**No placeholders remain.** An earlier draft left three `/* … */` holes in Task 5 for the arca harness's constructor names. Reading `lifecycle.rs` and `gascan-oci-fixture` to close them surfaced **three** constraints the holes were hiding, each of which would have failed the first run: the manifest needs `user = 'root'` (stock alpine has no `workspace` account), the image needs a rewritten `Cmd` (alpine's own exits immediately, so there would be no container left to `exec` into), and the network must be `networked` (offline is the refuted capability). That is the argument for filling holes rather than annotating them — the hole was not missing syntax, it was three missing facts. + +That discovery also changed Task 1's API: `for_image` takes a whole manifest rather than a network string, matching `policy_request_from_manifest(name, image, manifest)`, because `user` is a manifest fact and a network-only parameter cannot carry it. + +**Type consistency:** `CreateRequestFixture` is introduced in Task 1 with `pinned`/`for_image`/`request`, and used under exactly those names in Tasks 2, 4 and 5. `backend_contract(&dyn RuntimeBackend, &CreateRequestFixture)` is defined in Task 2 and called with that arity in Tasks 4, 5, 6 and 7. `capabilities()` is defined in Task 1 and used in Task 2. From 8c0bec5a1b96693457da01564b3f64bab9eaf54a Mon Sep 17 00:00:00 2001 From: Richard Kiene Date: Thu, 20 Aug 2026 11:20:40 -0700 Subject: [PATCH 3/3] docs: the assignment is P5.3, and the cold-start block says so PR #87 merged, so the cold-start block's headline was a decision already made. Replaced with the assignment the maintainer chose on 2026-08-20 -- P5.3, the backend conformance suite -- pointing at the design and the eight-task plan committed alongside this. The block carries the one fact that misleads a cold reader who skips the design: the roadmap's "extract the conformance suite from `fake_runtime.rs`" reads as an instruction to write one, and `crates/gascan-core/tests/backend_contract.rs:149` already **is** `pub async fn backend_contract(backend: &dyn RuntimeBackend)`. It is unreachable from the backend crates only because a `tests/` target is not a library, which is why `crates/gascan-apple/tests/live/backend_contract.rs` hand-rolls 65 lines over the same walk. Relocation and instantiation, not authorship. It also carries what the task is not -- the product-level `gascan-e2e`-on-arca work, P5.4/U5, offline -- because each is a plausible thing to widen into and each is named out of scope in the design with a reason. The flake and daemon-log queue is demoted from "the assignment" to what comes after it. Nothing in it changed. --- docs/status/START-HERE.md | 41 ++++++++++++++++++++++++++++++++------- 1 file changed, 34 insertions(+), 7 deletions(-) diff --git a/docs/status/START-HERE.md b/docs/status/START-HERE.md index 822ca93..4039f01 100644 --- a/docs/status/START-HERE.md +++ b/docs/status/START-HERE.md @@ -6,9 +6,9 @@ addressed to you, the agent. Follow it as instructions — there is nothing to p Rewritten 2026-08-18 after **MILESTONE 4 MERGED**, and updated the same day after the daemon instance record's publish race was fixed and merged (PR #80, #81). Updated again 2026-08-19, from branch `fix/daemon-reader-retryable-verdict`, after open item 1's residual — the reader's -retryable verdict — was implemented there and opened as PR #87. **Updated 2026-08-20: PR #87 IS -MERGED, open item 1 is closed entire, and the queue in the block below is now the whole -assignment.** Everything above the `Where the work is` heading is current; everything below it is +retryable verdict — was implemented there and opened as PR #87. **Updated 2026-08-20: PR #87 is +merged, open item 1 is closed entire, and the assignment is now P5.3 — the backend conformance +suite, specced and planned in the block below and not started.** Everything above the `Where the work is` heading is current; everything below it is history, **with seven exceptions that are current**: the sections headed `THE FOURTH MECHANISM` through `THE TENTH MECHANISM`, which describe live CI flakes — the ninth is about the *local* suite, so read it before trusting a green local run, and the tenth (added 2026-08-20) is why @@ -18,8 +18,35 @@ suite, so read it before trusting a green local run, and the tenth (added 2026-0 ## IF YOU READ NOTHING ELSE, READ THIS BLOCK -**PR #87 IS MERGED. OPEN ITEM 1 IS CLOSED ENTIRE, PRODUCER AND READER. THE QUEUE BELOW IS THE -WHOLE ASSIGNMENT — there is nothing left to decide about the reader half.** Merged 2026-08-20 as +**THE ASSIGNMENT IS P5.3, THE BACKEND CONFORMANCE SUITE. IT IS SPECCED, PLANNED, AND NOT STARTED.** +Chosen by the maintainer on 2026-08-20 after PR #87 merged, on the instruction to follow roadmap +order. Read these two, in this order, and nothing else is needed to begin: + +| | | +|---|---| +| Design | `docs/superpowers/specs/2026-08-20-backend-conformance-suite-design.md` | +| Plan | `docs/superpowers/plans/2026-08-20-backend-conformance-suite.md` — eight tasks, execute with `superpowers:subagent-driven-development` | + +**The one thing that will mislead you if you skip the design.** The roadmap says *"extract the +conformance suite from `fake_runtime.rs`"*, which reads as though a suite must be written. It must +not. `crates/gascan-core/tests/backend_contract.rs:149` is **already** +`pub async fn backend_contract(backend: &dyn RuntimeBackend)`. It cannot be reached from +`gascan-apple` or `gascan-arca` because it lives in a `tests/` target, which is not a library — +which is why `crates/gascan-apple/tests/live/backend_contract.rs` hand-rolls 65 lines over the same +ground. The task is relocation and instantiation, not authorship. + +**The measurement the whole task exists for is Task 5 step 6: run the contract against arca for the +first time.** The plan deliberately writes no expected result for it. If arca fails, the deliverable +is the failing test committed with the failure quoted — **not** a weakened assertion. That is +acceptance criterion 8 in the design, and it is the one that will be under pressure. + +**What is NOT in it, so nobody widens it mid-flight:** the product-level `gascan-e2e`-on-arca work +(P5's *second* exit clause), P5.4/U5, and anything about offline. Each is named under the design's +"Out of scope" with a reason. + +--- + +**PR #87 IS MERGED. OPEN ITEM 1 IS CLOSED ENTIRE, PRODUCER AND READER.** Merged 2026-08-20 as a true merge commit, not a squash: `7e84646`, parents `61f1b3c` (main) + `be19551` (branch), per `git log origin/main -2 --format='%h %p %s'`. All 35 commits survive (`git rev-list --count 61f1b3c..7e84646` → 35), and `git merge-base --is-ancestor` confirms @@ -83,8 +110,8 @@ eleven of its last twelve runs, since before this branch existed. insertions, markdown**. There is no code in it. A red `rust` there is a flake, exonerated by diff alone. Run `git diff --name-only` before you spend an hour on a red run. -**THE QUEUE BELOW IS THE ASSIGNMENT.** It is a queue, not a menu, -and the maintainer chooses from it: +**THE QUEUE BELOW IS WHAT COMES AFTER P5.3, NOT INSTEAD OF IT.** It is a queue, not a menu, and +the maintainer chooses from it once the conformance suite is done: 1. **The seven unfixed flake mechanisms this file names** — the empty pid-file read, the `reconcile` phase matrix that is red on `main` itself, the `lifecycle` ephemeral-port