P5.3: the backend conformance suite, and what it measured on each backend - #90
Merged
Conversation
`backend_contract` lived in `crates/gascan-core/tests/backend_contract.rs`,
a test target rather than a library, so no other crate could import it. It
moves to `gascan-conformance`, which apple and arca can both depend on.
The body is the original at `crates/gascan-core/tests/backend_contract.rs:149-179`
unchanged but for three things: the signature takes `fixture: &CreateRequestFixture`,
the `create_request("contract")` line goes away because the request is now
passed in, and one fully-qualified `gascan_core::runtime::ContainerState::Stopped`
shortens against the new import. `diff` of the two bodies shows those three and
nothing else. The fixture is a parameter because apple and arca pin different
images: `PolicyCompiler::compile` pins the approved workspace image, which a
live engine's seeded store does not hold.
The old copy in `gascan-core` is deliberately left in place; a later commit
removes it.
`tests/fake.rs` instantiates the contract against `FakeRuntime` through a
`&dyn RuntimeBackend`. It binds the fixture to a local before borrowing it,
so the fixture's `TempDir` outlives the request that points into it.
Verified at this tree:
- `cargo test -p gascan-conformance --test fake` before the move: exit 101,
E0432 unresolved import `gascan_conformance::backend_contract`.
- After the move: exit 0, 1 passed.
- Inverting the final assertion to
`assert!(backend.inspect(&id).await.unwrap().is_some())` and re-running:
exit 101, panic at `crates/gascan-conformance/src/lib.rs:124:5`. Reverted
and re-run: exit 0, and `cmp` against the pre-edit copy exits 0, so the
revert is byte-identical.
- `cargo fmt --all --check`: exit 0.
- `cargo clippy --workspace --all-targets -- -D warnings`: exit 0.
`gascan-core` is untouched: `git diff --quiet HEAD -- crates/gascan-core/`
exits 0. Its lint gate at `crates/gascan-core/src/lib.rs:2` is unchanged, and
`grep -rn "allow(clippy::\(expect_used\|panic\|unwrap_used\))" crates/` finds
no matches.
`crates/gascan-conformance` has held `backend_contract` since 9dcca6a, and until now `gascan-core` held a second copy of it. Delete the original. Removed from `crates/gascan-core/tests/backend_contract.rs`, line numbers as they stood at 9dcca6a: - `:149-181`, `pub async fn backend_contract(backend: &dyn RuntimeBackend)`. - `:653-658`, `fake_runtime_satisfies_backend_contract_through_trait_object`, its only caller. `grep -rn "backend_contract(" crates/` at 9dcca6a found exactly that one call site; `crates/gascan-conformance/tests/fake.rs` replaces both. `crates/gascan-core/tests/common/mod.rs` is deliberately unchanged. It is compiled into one target only -- `grep -rn "mod common;" crates/gascan-core/tests/` returns a single hit, `backend_contract.rs:1` -- and every fixture it exports still has callers among the fake-only tests that remain: 22 uses of `capabilities`, 26 of `create_request`, 10 of `create_request_with_network`. Nothing there became unused, so nothing there was deleted. No import in `backend_contract.rs` became unused either; `RuntimeBackend` now appears only on its `use` line but is still required to bring the trait's methods into scope. Also merges the two `use gascan_core::runtime::{...}` groups in `crates/gascan-conformance/src/lib.rs` into the one at `:10`. The second, at `:86-88`, was an artifact of keeping 9dcca6a append-only so its review could see the relocation was a relocation; that reason is spent. Verified: - `cargo test -p gascan-core` before the deletion: exit 0, 175 passed, 0 failed, 0 ignored. - `cargo test -p gascan-core` after: exit 0, 174 passed, 0 failed, 0 ignored. Delta is exactly 1, the removed trait-object test. No other test was deleted, weakened, or ignored, and none failed. - `cargo clippy -p gascan-core --all-targets -- -D warnings`: exit 0. - `cargo clippy -p gascan-conformance --all-targets -- -D warnings`: exit 0. - `cargo test -p gascan-conformance --test fake`: exit 0, 1 passed. - `cargo fmt --all --check`: exit 0.
crates/gascan-apple/tests/live/backend_contract.rs walked the same ground as
the shared suite in 65 hand-rolled lines. It now calls
gascan_conformance::backend_contract, keeping the one assertion the shared
contract does not make: that list_resources() reports nothing named for the
sandbox once it has been removed. gascan-conformance is a [dev-dependencies]
entry only, so it is compiled into no shipped artifact.
The test's name changed, so tests/ci/expected-ignored-tests.txt line 8 becomes
backend_contract::backend_contract_holds_on_apple. Before that edit
./scripts/ci-check-ignored-tests.sh exited 1, naming exactly that one removal
and one addition; after it, exit 0, "49 ignored test(s), matching the
baseline".
THE LIVE RUN FAILED. On host newcombe (Darwin 25.6.0 arm64, container CLI
1.1.0, service running) on 2026-08-20,
cargo test -p gascan-apple --test live -- --ignored backend_contract_holds_on_apple
exited 101:
panicked at crates/gascan-conformance/src/lib.rs:104:5:
assertion `left == right` failed
left: Running
right: Stopped
That is the shared contract's post-create state assertion. It is not a
regression from the extraction: the same assertion stands byte-identical at
daa687b^:crates/gascan-core/tests/backend_contract.rs:160-163, where it only
ever ran against FakeRuntime. Apple's create emits `container run`
(crates/gascan-apple/src/translate.rs:100), which starts the container, so
apple leaves the sandbox Running where the fake leaves it Stopped. Apple's
old file asserted nothing about state after create, so the divergence was
never covered. It is left unfixed here, and the contract is not softened to
accommodate it.
Because the run stops at that assertion, whether apple satisfies the rest of
the shared contract -- the exec walk in particular -- is still unmeasured.
The old file called start twice and stop twice to assert idempotence; the
shared contract calls each once. Apple's idempotence is therefore asserted
nowhere until a later task in this plan promotes it into the shared contract.
Residue from the failed run -- container
gascan-live-backend-92391-1787258495344035000-2e7e3b521ca5 and its three
gascan-{cache,mise,config}- volumes -- was removed by exact name; `container
list --all`, `container volume list` and `container network list` afterwards
match their pre-run output.
cargo test -p gascan-apple --no-run, cargo fmt --all --check and
cargo clippy --workspace --all-targets -- -D warnings each exited 0.
It fails, at the same assertion apple fails, with a third answer.
Run on host `newcombe`, 2026-08-20, against the engine `engine/arca-pin.json`
pins: revision c545612b056e028d5885968a7b9f586d694f994c, tag gascan-engine-m4.
GASCAN_ARCA_ENGINE_BIN=.../arca-engine \
GASCAN_ARCA_KERNEL_PATH="$HOME/Library/Application Support/dev.gascan/engine/vmlinux" \
GASCAN_ARCA_VMINIT_LAYOUT="$HOME/Library/Application Support/dev.gascan/engine/vminit" \
GASCAN_ARCA_BASE_OCI_LAYOUT=/tmp/alpine-oci \
cargo test -p gascan-arca --test live -- --ignored --test-threads=1 \
--nocapture backend_contract_holds_on_arca
Exit 101. Full output:
running 1 test
test conformance::backend_contract_holds_on_arca ...
thread 'conformance::backend_contract_holds_on_arca' (14162575) panicked at
crates/gascan-conformance/src/lib.rs:104:5:
assertion `left == right` failed
left: Creating
right: Stopped
note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace
FAILED
failures:
conformance::backend_contract_holds_on_arca
test result: FAILED. 0 passed; 1 failed; 0 ignored; 0 measured;
28 filtered out; finished in 1.16s
The failing assertion is the post-`create` state assertion at
`crates/gascan-conformance/src/lib.rs:104` -- the same one apple fails
(`0e1f3fb`). Everything the contract does after it -- `start`, `exec`, `stop`,
`remove`, and the final absent-`inspect` -- was NOT REACHED.
Reproducible: 4 runs on this host on 2026-08-20, all four `Creating` vs
`Stopped` at `lib.rs:104` (1.30s, 1.13s, 1.00s, 1.16s).
**`Creating` is terminal here, not a transient the contract read too early.**
MEASURED with a throwaway probe (created, then polled `inspect` every 200ms for
30s, printing only on change; not committed): the sole transition printed was
the first read, `PROBE 0.00s state=Some(Creating)`, and `PROBE final after 30s:
Some(Creating)`. The state never moved. So the failure is not a missing wait.
That makes three backends and three different post-`create` states for the same
compiled request: `FakeRuntime` reports `Stopped`, apple reports `Running`
because its `create` emits `container run` (`gascan-apple/src/translate.rs:96`),
and arca reports `Creating`. The assertion encodes the fake's answer and neither
real backend agrees with it, nor with the other. Recorded as an observation; the
contract is deliberately left unchanged, because a suite edited until it passes
measures nothing.
The test's own image is `engine.image(TAG)` and not the bare tag the plan's
sketch used. MEASURED on this host, with the bare tag: the run panicked at
`gascan-conformance/src/lib.rs:57` with `compile backend-contract policy:
InvalidWorkspaceImage` in 0.66s, before any call reached the backend --
`PolicyCompiler::compile_for_image` refuses a mutable reference
(`gascan-core/src/policy.rs:179`). `LiveEngine::image` supplies the store's
`repository@sha256:...`, which is what the tier's own `policy_request_from_manifest`
passes.
`tests/ci/expected-ignored-tests.txt` goes 49 -> 50 lines. The guard was run
before the baseline edit and FAILED exit 1 naming the added entry, then run
after it and passed: `ci-check-ignored-tests: 50 ignored test(s), matching the
baseline`, exit 0.
…m kill()
Fix round 1 on Task 5's review. `049b4ba` is not amended: it is cited by the
review document and the plan's ledger, and a correcting commit is the honest
evidence that the correction happened. No live test was re-run in this round;
the measurement recorded in `049b4ba` stands unchanged.
CORRECTION 1. `049b4ba` says the three backends give three post-`create` states
"for the same compiled request". **That is false.** The requests differ in four
inputs, each forced by the backend's own store and base image: apple's is
`CreateRequestFixture::pinned(&name, "offline")`
(`crates/gascan-apple/tests/live/backend_contract.rs:14`) -- pinned approved
workspace image, `network = offline`, default `workspace` user, nonce-suffixed
name. Arca's is `for_image("conformance", &engine.image(TAG), MANIFEST)`
(`crates/gascan-arca/tests/live/conformance.rs:48`) -- store digest reference,
`network = networked`, `user = root`, fixed name. What is held constant is the
CONTRACT (`crates/gascan-conformance/src/lib.rs:94-124`), not the request. The
finding survives; the sentence claimed a controlled comparison that was not one,
and would have had the next person rule the request out as a variable.
CORRECTION 2. `049b4ba` says "`Creating` is terminal here" and that arca's
`create` leaves the sandbox in `Creating` "indefinitely". **Both overclaim.**
`Creating` is not terminal -- `start` leaves it in under two seconds, which is
what the positive control does whenever it passes (`finished in 1.99s`, exit 0,
`.superpowers/sdd/2026-08-20-backend-conformance-suite/arca-positive-control.log`),
and "indefinitely" extrapolated past the 30s observed. The defensible claim is
narrower and carries the whole finding: **`create` performs no autonomous state
transition; the sandbox sits in the engine's `created` status until something
starts it.**
That claim is now anchored to the pinned engine's own source rather than to a
deleted probe. Verified here: `git -C .artifacts/arca-engine/arca rev-parse HEAD`
-> c545612b056e028d5885968a7b9f586d694f994c, matching `engine/arca-pin.json:6`,
with only the generated `Sources/ContainerBridge/BuildInfo.generated.swift`
modified. At that revision `Sources/ArcaEngine/EngineTranslation.swift:127-134`
maps `case "created": return .creating` (`:129`). It is a total function of the
current status string with no time term, so no wait changes what it returns --
decisive by construction. The probe is demoted to corroboration: it was never
committed and its source is quoted nowhere, so its single line of output is
indistinguishable from what a broken probe would print.
CODE CHANGE. `engine.kill().await` is now the last statement of the test, with a
comment explaining it, following `connect.rs:96-103`'s shape for the tier's one
other exception. The earlier rationale for omitting it -- that `kill()` adds an
unrelated failure mode -- is withdrawn; it is exactly the argument
`crates/gascan-arca/tests/live/common/mod.rs:473-477` rejects ("This assertion is
what stops that regressing, and it is here rather than only in `shutdown.rs`
because every test in this tier stops an engine"), guarding an abort measured at
6 crashes in 192 runs before the engine fix (`:462-471`).
**The added line does not execute today and changes no recorded result.**
`backend_contract` panics at `crates/gascan-conformance/src/lib.rs:104`, so
control never reaches the next statement, and no statement before the contract
call is touched by this diff. That is an argument from unreachability, not a
re-measurement: the live test was not re-run.
Also corrected, a wrong anchor in `049b4ba`: apple's `create` emits `container
run` at `crates/gascan-apple/src/translate.rs:100`, inside the `create` that
opens at `:94`. `:96`, which `049b4ba` cites, is `validate_view(&view)?;`. The
claim was true; the pointer was not.
Also added: a comment on the staying-up `Cmd` (the one deliberate difference
from apple that carried none), pointing at `lifecycle.rs:33-53`.
Verified after the edit on `newcombe`, 2026-08-20:
`cargo test -p gascan-arca --test live --no-run` exit 0;
`cargo fmt --all --check` exit 0;
`cargo clippy --workspace --all-targets -- -D warnings` exit 0.
The ignored-test baseline is unchanged at 50 lines; this round adds and removes
no test, and the guard was not re-run.
`duplicate_create_is_rejected_and_start_stop_are_idempotent` in
crates/gascan-core/tests/backend_contract.rs held three assertions that any
`RuntimeBackend` owes, stated against `FakeRuntime` alone. All three move into
`backend_contract`: the doubled `start`, the doubled `stop`, and a second
`create` of a held id failing with `resource_conflict`.
Duplicate-create promoted too, against the brief's default of leaving it behind,
because it clears the design's §4 bar. It reaches no fake-only machinery -- no
`calls()`, `outcomes()`, `seed_*`, `FailureBoundary`, or `persistent` -- and
each backend detects the collision its own way: the fake by map lookup
(crates/gascan-core/src/fake_runtime.rs:695), apple by a pre-flight inventory
scan (crates/gascan-apple/src/backend.rs:238), arca engine-side, its wire code
mapped at crates/gascan-arca/src/error.rs:49. Only the stable code is portable,
so only the stable code is asserted. Because both halves promoted, the whole
test function goes rather than half of it: `cargo test -p gascan-core` was
174 passed / 15 suites before and is 173 passed / 15 suites after, both exit 0.
These assertions are exercised by `FakeRuntime` only today. Apple and arca still
fail the contract at the post-`create` state assertion
(crates/gascan-conformance/src/lib.rs:104), which precedes every line added
here, so neither reaches a doubled `start`, a doubled `stop`, or the second
`create`. Task 4 recorded apple's idempotence as asserted nowhere until a later
task promoted it; this is that task, and the assertion now exists in the shared
contract for all three backends -- but no idempotence or duplicate-create
behaviour has been measured on apple or arca. No live test was run.
Each promoted assertion was mutation-checked against `FakeRuntime`, the file
`touch`ed first so the run recompiled rather than serving a cached artifact
(each failing run's output carries `Compiling gascan-core v0.1.20`):
- `start` rejecting a second call -> exit 101, panic at lib.rs:116:30,
the second `backend.start(&id)`.
- `stop` rejecting a second call -> exit 101, panic at lib.rs:127:29,
the second `backend.stop(&id)`.
- the fake's duplicate-id conflict check deleted -> exit 101, panic at
lib.rs:109:49, `unwrap_err()` on an `Ok` value.
Every mutation was reverted and crates/gascan-core/src/fake_runtime.rs is
byte-identical to its pre-mutation state -- sha256
14c289767b0440c99c9b0ab0e4ff82a315dba73f6dc05473123fb4b24186e827 before and
after, `cmp` against a pristine copy clean, and the file is absent from this
commit. `cargo test -p gascan-conformance --test fake` exits 0 after the revert.
`cargo fmt --all --check` exit 0, `cargo clippy --workspace --all-targets --
-D warnings` exit 0, `./scripts/ci-check-ignored-tests.sh` exit 0 at 50 ignored
tests, unchanged.
…e anchors Two review fixes on top of 22dfee8. No assertion was weakened, added, or reordered; the post-`create` state assertion at crates/gascan-conformance/src/lib.rs:104-107 is untouched, and crates/gascan-core/src/fake_runtime.rs is still sha256 14c289767b0440c99c9b0ab0e4ff82a315dba73f6dc05473123fb4b24186e827. I-1, a record correction. Every line anchor in 22dfee8's message is four lower than the tree that commit ships, because the four-line comment block at lib.rs:108-111 was added after the mutation cycles ran and the numbers were recorded during them. The runs and their panics are exactly as reported; only the pointers were stale. Anchored three ways: assertion during the runs in 22dfee8 in this commit second `start` 116:30 120:30 136 second `stop` 127:29 131:29 147 `unwrap_err()` 109:49 113:49 112 Verified by `git show 22dfee8:crates/gascan-conformance/src/lib.rs | grep -n`, which puts `unwrap_err` at 113 and the doubled calls at 119/120 and 130/131. The "in this commit" column names the call sites only -- no mutation was run against this tree, so no panic is claimed at those lines. `git show 22dfee8:crates/gascan-conformance/src/lib.rs` is the durable way to read the middle column. I-2, a latent leak closed. The promoted duplicate-`create` assertion kept only `.code()` and dropped the `CreateFailure`, which carries whatever resources the rejected create had already built; the walk's `remove` covers the first create's resources only. Nothing was observed leaking -- this prevents a leak rather than fixing one. It is a no-op on all three backends today: the fake's map lookup (crates/gascan-core/src/fake_runtime.rs:695) and apple's pre-flight inventory scan (crates/gascan-apple/src/backend.rs:238-252) both fire before a single resource is made, and arca never reaches the line. It exists for live engines, where a conflicting create has been measured reporting the three volumes it had made (crates/gascan-arca/tests/live/lifecycle.rs:275-283). The failure is now bound and its `created()` removed when non-empty. Nothing is asserted about that list. `is_empty()` would be a new portability claim, and arca's own live suite predicts arca would fail it -- asserting something we expect to fail and have not measured is the opposite of what this branch is for. Because removal is cleanup and not a claim, there is no assertion here for a spec §6 mutation to falsify, so no mutation proof accompanies this change; the three proofs covering the promoted assertions stand unchanged from 22dfee8. `cargo test -p gascan-conformance --test fake` exit 0, 1 passed. `cargo test -p gascan-core` exit 0, 173 passed across 15 suites, and its backend_contract suite 33 passed -- both unchanged from 22dfee8, as expected for a change confined to gascan-conformance. `cargo fmt --all --check` exit 0. `cargo clippy --workspace --all-targets -- -D warnings` exit 0. No `#[ignore]` added or removed: `git diff | grep -c ignore` is 0, so the 50-line baseline is untouched. No live test was run.
…sandbox
Reverses the `if !conflict.created().is_empty() { ... remove ... }` block added
in 9f2c0bf. The review finding it answered, and the ruling that directed it,
both rested on a false premise: that `conflict.created()` names orphaned
residue. It cannot. Both creates in this walk pass the same `fixture.request()`
-- crates/gascan-conformance/src/lib.rs:97 and :112 -- so the same sandbox id
and the same resource names. Anything the rejected create reported would
therefore carry the names of the live sandbox the walk still has to `start`,
`exec`, `stop` and `remove`, and removing it would have torn that sandbox down
mid-walk and double-removed its container and managed network at the closing
`RemoveRequest::from_resources(created.created().to_vec())`.
The live measurement the premise leaned on does not transfer.
crates/gascan-arca/tests/live/lifecycle.rs:259-278 removes the container and the
three volumes first, keeping only the network name held, and only then does the
conflicting create rebuild those three volumes and fail -- so there they are
genuinely orphaned. Here "residue" and "the sandbox under test" are the same
names, so what is cleanup in that scenario is a teardown in this one.
Nothing was observed misbehaving. The branch never fired on any backend: the
fake's map lookup and apple's pre-flight inventory scan both reject the
duplicate before building anything, and apple and arca do not reach this line at
all. This removes latent wrong code rather than fixing an observed failure. Two
latent surfaces go with it -- the mid-walk teardown and double-remove above, and
the `.unwrap()` on that removal, which on a live engine reporting residue it had
already rolled back would have unwrapped a remove of something absent, whose
error status is backend-defined.
No narrower cleanup replaces it. Separating "residue" from "the sandbox under
test" would need a distinction nobody has measured on any live backend. In its
place the code records the open question with its anchor, at lib.rs:114-123.
The assertion keeps 9f2c0bf's bound form -- `let conflict = ...;
assert_eq!(conflict.code(), "resource_conflict");` -- rather than folding back
to the chained expression 22dfee8 shipped. It is two lines instead of four, and
the comment that replaces the removed block is about `conflict.created()`, so
the value it discusses should be named in the code beside it.
9f2c0bf's I-1 anchor corrections stand unchanged. Its comment cited
`gascan-apple/src/backend.rs:238-252` for apple's pre-flight scan; that text is
deleted here, so the correction is moot, but for the record `:252` is the
closing brace of the conflict block and `:253` is `let mut created =
Vec::new();` -- verified by numbering the file -- so the range as written ended
correctly and it is `:253`, not `:252`, that is the first line after the scan.
`cargo test -p gascan-conformance --test fake` exit 0, 1 passed. `cargo test -p
gascan-core` exit 0, 173 passed across 15 suites, backend_contract 33 passed --
both unchanged from 22dfee8 and 9f2c0bf. `cargo fmt --all --check` exit 0.
`cargo clippy --workspace --all-targets -- -D warnings` exit 0.
crates/gascan-core/src/fake_runtime.rs is still sha256
14c289767b0440c99c9b0ab0e4ff82a315dba73f6dc05473123fb4b24186e827 and absent from
this commit. No `#[ignore]` added or removed. No live test was run. This round
removes code and adds no assertion, so no mutation proof is owed.
Promotes one assertion out of
`exec_session_is_live_bidirectional_and_emits_one_exit` in
crates/gascan-core/tests/backend_contract.rs into `backend_contract`: after the
terminal `ExecOutput::Exit`, the session yields nothing further. It is now
crates/gascan-conformance/src/lib.rs:148 and the duplicated half is gone from
gascan-core.
It clears the design's §4 bar on both halves. Each backend closes the stream in
its own code -- the fake's spawned task returns after the `Exit` send so
`outputs` drops (crates/gascan-core/src/fake_runtime.rs:1123-1127), apple breaks
on `terminal` after mapping `AttachOutput::Exit`
(crates/gascan-apple/src/backend.rs:604 and :614-616), arca breaks on the same
flag over engine frames (crates/gascan-arca/src/backend.rs:368-374 and
:387-389). And it is load-bearing rather than incidental: gascand drains to the
end of the stream in two places, crates/gascand/src/service.rs:2336 and
crates/gascand/src/ssh/manager.rs:700, each with a second drain after
`session.cancel()` (:2341 and :707). A backend that kept the stream open after
`Exit` hangs all four forever and nothing else in the suite would notice. It
reaches no fake-only machinery: no `calls()`, `outcomes()`, `seed_*`,
`FailureBoundary`, or `FakeRuntime::persistent`.
The rest of that test does not promote and stays. The stdin echo needs the
fake's own `fake-echo-stdin` (fake_runtime.rs:596), which no container image
has, and `Exit { code: 143, signal: 15 }` is the fake's `128 +
signal` arithmetic (fake_runtime.rs:1118-1122) -- apple hardcodes `signal: 0` on
every exit (gascan-apple/src/backend.rs:604) and arca passes the engine's
`exit.signal` through (gascan-arca/src/backend.rs:370-373), so that is three
behaviours rather than one contract. The test is renamed
`exec_session_echoes_stdin_and_maps_a_signal_to_its_exit_code` because
"emits_one_exit" was the name of the assertion that left, and a doc comment says
where it went.
This assertion is exercised by `FakeRuntime` only today. Apple and arca still
fail the contract at the post-`create` state assertion
(crates/gascan-conformance/src/lib.rs:104) -- apple reports `Running`, arca
`Creating` -- which precedes the exec entirely, so neither backend reaches it and
no stream-termination behaviour has been measured on either. The three-backend
evidence above is code read statically, not a measurement. No live test was run.
Mutation-checked against `FakeRuntime`, the file `touch`ed first so the run
recompiled rather than serving a cached artifact. Mutation: a second
`send_fake_exec_output(.., ExecOutput::Exit { code, signal })` appended after the
terminal one, so the stream no longer ends there. `cargo test -p
gascan-conformance` exited 101, its output carrying `Compiling gascan-core
v0.1.20`, and it panicked at crates/gascan-conformance/src/lib.rs:148:5,
`assertion failed: session.next().await.is_none()`. The comment block above the
assertion was written before the cycle ran, not after, so that anchor is the
line this commit ships.
The mutation was reverted and crates/gascan-core/src/fake_runtime.rs is
byte-identical -- sha256
14c289767b0440c99c9b0ab0e4ff82a315dba73f6dc05473123fb4b24186e827 before and
after, `cmp` against a pristine copy exit 0, and the file is absent from this
commit.
`cargo test -p gascan-conformance` exit 0, 1 passed. `cargo test -p gascan-core`
exit 0, 173 passed across 15 suites -- unchanged from e7e55e4, because this moves
an assertion rather than a test function. `cargo fmt --all --check` exit 0. No
`#[ignore]` added or removed: the diff contains no `#[ignore]` line in either
direction, and the renamed test carries none and appears nowhere in
tests/ci/expected-ignored-tests.txt.
§3 loses its "Estimated 6-8 will promote cleanly" and gains the measured outcome. Across Tasks 6 and 7, FOUR assertions promoted: three in 22dfee8 (the doubled `start`, the doubled `stop`, a second `create` of a held id failing with `resource_conflict`) and one in a32a29e (an exec session's terminal `Exit` ends the stream). Counted by candidate rather than by assertion, one of the seven named tests promoted whole, one promoted in part, and five promoted nothing. Both countings fall below 6. Nothing was promoted to move the number; the estimate is corrected instead, beside the note that an earlier draft said 13. A fake-only table now names each remaining candidate and the machinery that decided it: `set_exec_result` and `set_logs`; `seed_volume`; `FakeRuntime::persistent`. Two entries are not blocked by machinery at all but by the contract's shape -- it is one walk over one fixture, and arca's must be `network = 'networked'`, so `offline_fake_create_has_no_managed_network` and the network element of the ordering assertion would each need the contract to branch on the fixture. That is a design change and this task does not make one. Two corrections carried in, both re-derived at a32a29e rather than repeated: - The plan's candidate table says `networked_fake_create_reports_network_then_volumes_then_container` asserts its ordering "through the fake's call recorder". It does not. It reads `outcome.created()` at crates/gascan-core/tests/backend_contract.rs:509-517 and touches neither `calls()` nor `outcomes()`, and `CreateOutcome::new` stores the vec verbatim (crates/gascan-core/src/runtime.rs:778-784). The verdict is unchanged, so the wrong reason would have survived unnoticed. The reasons that do hold are recorded in its place. - §2 now states that `crates/gascan-core/tests/common/mod.rs` does NOT go away and the duplication with `gascan-conformance` is permanent, not "deliberate and short-lived" pending a "Task 9". The plan has eight tasks and there is no Task 9; the false sentence is at .superpowers/sdd/2026-08-20-backend-conformance-suite/task-3-brief.md:21 and was repeated in that task's report, never in this design -- so it is stated correctly here rather than rewritten. MEASURED at a32a29e: `mod common;` appears in exactly one file, backend_contract.rs:1 (policy.rs defines its own `capabilities()` at :19), and that one target still calls `capabilities()` on 20 lines, `create_request(` on 24 and `create_request_with_network(` on 9, so every export survives. Task 3's Step 2 deferred this deletion check to the nonexistent Task 9; it was run here -- `grep -rn "create_request\|capabilities()\|CreateRequestFixture" crates/gascan-core/tests/` returns 63 lines, exit 0 -- and nothing is unreferenced, so common/mod.rs is unchanged. §3's opening count is re-anchored too: 23 `#[tokio::test]` functions was true at 10e3342 and is 21 at a32a29e. Documentation only; no code changes. `cargo fmt --all --check` exit 0.
All three backends report a different state after `create`, and only the test double satisfies the contract. `FakeRuntime` reports `Stopped`, apple `Running` (its `create` compiles to `container run`, translate.rs:100), arca `Creating` (the pinned engine maps status "created" -> .creating). Apple and arca both panic at gascan-conformance/src/lib.rs:104, the walk's third assertion -- so `start`, `exec`, `stop`, `remove` and the closing absent `inspect` were NOT REACHED on either real backend. They are unmeasured, not passing and not failing. docs/evidence/2026-08-20-backend-conformance.md is the durable record: the command and exit code for each backend, the host and date, the engine revision, the four arca reproductions, the positive control run BEFORE the conformance test existed, and the apple residue check. It also carries what this branch surfaced and deliberately did not answer -- what a same-request duplicate `create` reports in `created()`, unmeasured on every backend, with the cleanup that assumed otherwise written in 9f2c0bf and reversed in e7e55e4 -- and the nearest precedent a reader will find for that mistake, apple's tests/live/storage.rs:22-37, which is correct where it is. CORRECTS THE DESIGN'S §6, which claimed the arca instantiation "inherits CI coverage free". CI's live-tier step sets one variable and the tier needs four; `base_oci_layout()` panics rather than skips; apple's tier runs in no CI job at all, line 178 being the file's only `--ignored`. Both real-backend measurements are therefore local-only, on a named machine on a named date, and that is the only evidence for them that will ever exist. The new §6 states this as a derivation and says in terms that no CI run of the arca test has been observed. Also retitles §3's "Promote -- assertions any backend must satisfy" to "Candidates considered": five of the seven listed promoted nothing, and the measured table below it already corrected the substance. START-HERE records P5.3 as executed, points at the evidence, and opens item 10 for the two failures -- naming the assertion, both mechanisms, the three live design candidates, and the instruction not to close it by widening lib.rs:104. It does NOT claim P5's exit is met: the second clause, gascan-e2e on arca, is untouched and was out of scope. VERIFIED at this tree, each command run alone with the machine clear of other cargo jobs: `cargo fmt --all --check` exit 0; `cargo clippy --workspace --all-targets -- -D warnings` exit 0; `cargo test --workspace` exit 0, zero occurrences of FAILED / "error: test failed" / panicked across the whole log, 50 ignored; `./scripts/ci-check-ignored-tests.sh` exit 0, "50 ignored test(s), matching the baseline". `cargo tree -p gascan --edges normal` and the same for gascand return no gascan-conformance, so the crate is reachable from neither shipped binary. An earlier run of the same list failed `gascan --lib` on daemon::tests::inherited_startup_diagnostic_survives_path_replacement, which START-HERE:3232 names as 7 of the 12 failures in 43 runs of that command. Alone: `test result: ok. 1 passed; 0 failed; 323 filtered out; finished in 0.21s`, exit 0, against the 60s bound it had blown. No file under crates/gascan is touched by this branch. No live test was run here. No assertion edited, no #[ignore] added or removed.
Final whole-branch review, one Important and eight Minors, all comment or prose. No assertion, no #[allow], no #[ignore], no baseline change. Proof rather than assertion: `git diff -U0 -- crates/ | grep "^[+-]" | grep -v "^[+-][+-]" | grep -v "^\s*[+-]\s*//" | grep -v "^[+-]$"` returns nothing -- every added and removed line under crates/ is a comment or a blank. expected-ignored-tests.txt is untouched at 50 lines and fake_runtime.rs is not in the diff. I-1. The post-`create` state assertion, the one line two of three backends fail and acceptance criterion 8 forbids weakening, carried no comment; apple's instantiation carried none at all, against its arca sibling's fifteen lines on an unreachable kill(). Someone arriving from `panicked at ... left: Running, right: Stopped` had nothing at either the panic site or the test file saying it is a recorded result, and the cheapest path from there is to "fix" it. Three places now say so: the assertion itself, apple's test file, and lib.rs's module doc, which said why the crate exists but not what it currently measures. THAT MOVED THE ASSERTION OFF LINE 104. It is lib.rs:139 now. The recorded panic text stays verbatim -- it is a measurement, not a pointer -- and every prose citation of :104 was re-derived, with the reconciliation written in four places. `grep -rn "lib.rs:104" docs/` returns exactly one hit, the quoted panic. Two other shifted anchors corrected the same way: the two create calls :97/:112 -> :117/:147, the collision comment :114-123 -> :149-164. M-1. Apple's own list_resources tail (backend_contract.rs:37-44, re-derived after this wave) is unreached like everything after the panic and was missing from the evidence document's enumeration, two sections from a residue check run by hand against the CLI. Both are named now, and the difference stated. M-2. The collision comment warned against a shape without naming the precedent. It now names gascan-apple/tests/live/storage.rs:22-37 and says why that helper is correct where it is: its callers face independently-seeded state, not a duplicate of a live sandbox. M-3. The two permanent fixture copies now reference each other, citing the design's §2 for the permanence and the dev-dependency cycle for the reason. The (1,1,0)/(1,0,0) divergence is written down with its inertness argument rather than left to be re-derived: `grep -n "\.version" gascan-core/src/policy.rs` returns :422 constructing PolicyError::OfflineUnsupported and :561-563 formatting it, and :417-425 shows :422 is reachable only through NetworkIsolation::Unsupported, which neither copy sets. M-4. START-HERE declared P5.3 executed a few thousand lines above a line still assigning it the sandbox_id-claim sharing. Settled at both ends: struck through at the line itself with the reason (a production change, excluded wholesale by §5 -- `git diff --name-only main...feat/backend-conformance-suite | grep "/src/"` returns only crates/gascan-conformance/src/lib.rs), and restated in item 10, which points at the history by its text rather than by a line number. M-5. "the only `--ignored` in the file" was overstated in three artifacts. `grep -n -- "--ignored" .github/workflows/ci.yml` returns :139 and :162 inside comments and :178 executed. All three now say the only EXECUTED one. M-6. 0e1f3fb's message says the three `container ...` listings "afterwards match their pre-run output" and no pre-run listing was ever captured. The evidence document did that service for 049b4ba and not for this. It does now, quoting the sentence and saying what was actually measured and what it is not enough to claim. M-7. The CI comment's "20 of the 25" is from an earlier revision and both the evidence document and design §6 propagated it as current. Re-derived: `grep -rc '#\[ignore' crates/gascan-arca/tests/live/*.rs` sums to 29, corroborated by the arca run's own "28 filtered out" beside its 1 failure. Both documents now say so. M-8. The idempotence comment sat 22 lines from the stop pair it also governs; a one-line marker closes it. M-9 and M-10 left, as the review recorded them: seen and not worth the churn. AUDIT-4, correcting an anchor in 99f1449 rather than amending it, since the branch's rule is that a cited commit is not rewritten. 99f1449 cites START-HERE:3232 for "7 of the 12 failures in 43 runs". :3232 is a row of the supporting table; the sentence naming inherited_startup_diagnostic_survives_path_replacement as the dominant flake is at :3236 and the 43-runs total at :3235. VERIFIED at this tree, each run alone with exit codes from plain $? on unpiped commands: `cargo fmt --all --check` exit 0; `cargo clippy --workspace --all-targets -- -D warnings` exit 0, "No issues found". `cargo test --workspace` deliberately NOT run -- nothing here changes executable code, which the comment-only proof above establishes, and clippy --all-targets compiles every target touched. No live test was run.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
P5.3 of the Arca integration roadmap: make the existing
RuntimeBackendconformancecontract importable, and run it against all three backends so P5's first exit clause can
be measured rather than assumed.
Design:
docs/superpowers/specs/2026-08-20-backend-conformance-suite-design.mdPlan:
docs/superpowers/plans/2026-08-20-backend-conformance-suite.mdEvidence:
docs/evidence/2026-08-20-backend-conformance.mdThe headline: all three backends disagree, and only the double satisfies the contract
createFakeRuntimeStoppedRunningcreatecompiles tocontainer run(crates/gascan-apple/src/translate.rs:100)Creating"created"→.creatingBoth real backends fail at
crates/gascan-conformance/src/lib.rs:104, the post-createstate assertion. That assertion lived in
gascan-core/tests/where it only ever ranagainst the fake; apple's old hand-rolled file and arca's
lifecycle.rseach assertnothing about state between
createandstart, so nothing on any side ever covered thedivergence. This is the hole P5.3 existed to find.
It is not a regression from the extraction: the assertion stands unchanged, apart from an
import-shortened path, at
daa687b^:crates/gascan-core/tests/backend_contract.rs:160-163.Everything after that assertion —
start,exec,stop,remove, the final absentinspect, and apple's ownlist_resourcestail — was NOT REACHED on either backend.Nothing here says apple or arca passed or failed the exec walk.
Per the design's acceptance criterion 8, the two failing tests are committed unweakened.
The fix is separate work; forcing green by relaxing the assertion is the one outcome that
would make the exercise worthless.
Where each backend was run, and it is asymmetric
cargo test --workspace— CI covers it every pushRunningvsStoppednewcombe, 2026-08-20. No CI job runs apple's live tier at allCreatingvsStoppedc545612b. CI's live step sets one of the four variables the tier requires, so the test panics there before reaching the backendNeither real-backend measurement is reproducible in CI, which is why both are recorded
with their machine and date. The design's §6 previously claimed arca "inherits CI coverage
free"; that claim is corrected in this PR.
Two things make the arca result trustworthy rather than merely observed:
attributable to arca and not the machine:
lifecycle::create_start_inspect_stop_and_remove_drive_a_real_containerpassed, exit 0, in 1.99s.
Creatingis not the contract reading too early, and this is anchored to the engine'sown source rather than to a probe. At the pinned revision,
EngineTranslation.swift:127-134is a bare
switchover the status string withcase "created": return .creating— noclock, no retry, no stored history, so waiting cannot change what it returns.
createperforms no autonomous state transition; the sandbox sits in the engine's
createdstatusuntil something starts it.
What changed
gascan-conformance, dev-dependency only, compiled into no shipped artifact(
cargo tree --edges normalforgascanandgascandshows it in neither). It existsbecause
gascan-core/src/lib.rs:2deniesclippy::unwrap_usedand a conformance suite isbuilt from unwrapping assertions — the denial is untouched and no
#[allow]was addedanywhere.
verified mechanically against the pre-move file, so the diff shows a move and not a rewrite.
gascan-appleis deleted and replaced by a callinto the shared suite, keeping its one apple-local assertion.
start,doubled
stop, duplicate-create→resource_conflict, and the exec stream ending at itsterminal
Exit. Each was proved real by breaking the behaviour in the fake and watching thesuite fail at a named panic site, then reverting —
fake_runtime.rsis byte-identicalacross the branch.
fake-only machinery that blocks it. The estimate was 6-8 promotions; the measured outcome is
4, and the spec was corrected rather than the work stretched to reach the number.
Verification, against CI's own step list
cargo fmt --all --check0 ·cargo clippy --workspace --all-targets -- -D warnings0 ·cargo test --workspace0 (0 failures, 50 ignored) ·./scripts/ci-check-ignored-tests.sh0("50 ignored test(s), matching the baseline").
The first
cargo test --workspaceattempt failed ondaemon::tests::inherited_startup_diagnostic_survives_path_replacement. It was exonerated bydiff plus isolation, never by probability: the branch touches no file under
crates/gascan/, and the test passed alone in 0.21s against the 60s bound it had blown. It isTHE NINTH MECHANISM's dominant flake.
Open questions this branch surfaced and deliberately did not answer
RuntimeBackendowes aftercreate. Three backends give three answers; thecontract currently encodes the double's. Deciding it is a design question, not a bug fix.
createreports. In the contract's walk both creates usethe same request, so a collision's resources carry the live sandbox's own names — not
orphaned residue. A cleanup was written and reversed for that reason. Unmeasured on every
backend; the place to measure it is arca's live tier.
crates/gascan-apple/tests/live/storage.rs:22-37holds a helper in exactly the shapethat was reversed above. It is correct there and wrong in the walk, and the contract now
says so at the call site.
Out of scope by design, and untouched: the product-level
gascan-e2e-on-arca work (P5'ssecond exit clause), P5.4/U5, and offline. P5's exit is not met — this covers the
first clause only.