Fail fast in conformance E2E when stale emulator processes are alive (cross-talk hang)

[JakeMG-INL]

Context

Surfaced by the /audit-issue review of #52. Not a defect in #52's diff — an environment/test-infra hazard exposed while running the conformance E2E during that audit.

What happened

Two orphaned --auto-rearm emulator processes (run_secc.py / run_evcc.py), left over from a manual live-verify run, were still cycling SLAC/SDP on the shared virtual interface. The conformance E2E (tests/conformance/e2e) spins up its own run_secc/run_evcc subprocess pairs, which communicate over the same link-local multicast. The orphans and the E2E's pairs heard each other's frames, so every handshake collided and retried until timeout. A suite that normally finishes in ~60s sat stuck on a single scenario for 12+ minutes with no diagnostic.

Root cause is the shared virtual interface coupling plus the fact that an --auto-rearm process never self-exits (ADR-0005: only q quits), so a backgrounded one is immortal until killed.

What to build (in priority order)

1. Fail-fast pre-flight in the E2E fixture (core, fully specified). In tests/conformance/e2e/conftest.py, assert at session start that no run_secc.py/run_evcc.py processes are already alive (e.g. pgrep -f). If any are, fail immediately with a clear message listing the stale PIDs, instead of silently degrading into multi-minute timeouts.
2. Per-test wall-clock bound (defense-in-depth). Add a pytest-timeout cap (~30-60s/test) so a contended handshake aborts with a traceback rather than hanging indefinitely.

Optional / heavier follow-up (design decision — triage before doing)

3. Per-run transport isolation. Run each E2E pair in its own network namespace / unique veth pair or multicast group so orphans on the default interface can't be heard at all. Removes the failure mode entirely rather than detecting it; larger lift — split into its own issue if pursued.

Acceptance criteria

• Running the E2E with a stale run_secc.py/run_evcc.py process alive fails within seconds with a message naming the offending PID(s), instead of timing out.
• A clean E2E run is unaffected (still ~60s, 14 passed).
• (If Make AcCCS usable with different hardware setups #2 done) a hung scenario aborts at the per-test timeout with a traceback.

Related

• Parent/sibling: Re-assert EV presence (CP State B / SECC relay) on session re-arm #52 (re-assert EV presence on re-arm). This is test-infra hardening, not part of Re-assert EV presence (CP State B / SECC relay) on session re-arm #52's behavior change.

[JakeMG-INL]

This was generated by AI during triage.

Agent Brief

Category: enhancement
Summary: Make the conformance E2E layer fail fast (seconds, with a clear message) when stale run_secc.py / run_evcc.py processes are already alive, and bound each E2E test with a wall-clock timeout, instead of degrading into multi-minute cross-talk hangs.

Scope: items 1 (fail-fast pre-flight) and 2 (per-test timeout) from the issue body. Item 3 (per-run transport isolation) is split out — see the linked follow-up issue.

Current behavior:
The conformance E2E layer spawns its own run_secc / run_evcc subprocess pairs (via the launch_emulator factory fixture) that communicate over the shared acccs_secc/acccs_evcc veth link-local multicast. If an orphaned --auto-rearm emulator from a prior manual run is still cycling SLAC/SDP on that same interface, its frames collide with the E2E pairs' handshakes. Every handshake retries until timeout, so a suite that normally finishes in ~60s can sit on a single scenario for 12+ minutes with no diagnostic. An --auto-rearm process never self-exits (ADR-0005: only q quits), so a backgrounded one survives until killed.

Desired behavior:

1. Fail-fast pre-flight. Before any E2E test that spawns emulators runs, detect any already-alive run_secc.py / run_evcc.py process not owned by the current test session. If any exist, abort immediately with an error that names the offending PID(s), rather than letting tests degrade into timeouts. The check must be scoped so the codec and state-machine layers (which spawn no emulators) are unaffected — a developer without the veth pair / without stale processes sees no behavior change. The existing session-scoped veth_pair gate is the natural coupling point: only the E2E layer depends on it.
2. Per-test wall-clock bound. Each E2E test should abort with a traceback after a bounded interval (~30–60s) rather than hanging indefinitely. Use pytest-timeout; add it to requirements.txt (and it must be present in the AcCCS-box Pi conda env used by the Pi conformance gate). Pick a cap comfortably above a clean scenario's real runtime so healthy runs never trip it.

Key interfaces:

• launch_emulator factory fixture and the session-scoped veth_pair fixture in the conformance conftest.py (tests/conformance/conftest.py — note: the issue body's tests/conformance/e2e/conftest.py path is wrong; there is no e2e-level conftest, the fixtures live one level up). Add the pre-flight there as a session-scoped (autouse-on-E2E) check or fold it into the veth_pair gate.
• Detection mechanism: pgrep -f run_secc.py / pgrep -f run_evcc.py (pgrep is available). Must exclude the test session's own spawned children so the check doesn't flag the very processes launch_emulator is about to start / has started.
• pytest-timeout config in pytest.ini (per-test timeout= and method) plus the requirements.txt entry.

Acceptance criteria:

• Running the E2E with a stale run_secc.py / run_evcc.py process alive fails within seconds with a message naming the offending PID(s), instead of timing out.
• A clean E2E run is unaffected — still ~60s, all E2E scenarios pass (14 passed baseline).
• The pre-flight does not fire on the E2E's own launch_emulator-spawned children (no false positive mid-suite).
• Codec and state-machine layer tests (which don't use launch_emulator/veth_pair) are unaffected and still run without the veth pair present.
• A hung scenario aborts at the per-test pytest-timeout cap with a traceback rather than hanging indefinitely.
• pytest-timeout is added to requirements.txt.

Out of scope:

• Per-run transport isolation (network namespace / unique veth / distinct multicast group) — split into its own issue.
• Changing --auto-rearm self-exit behavior (ADR-0005 stands: only q quits).
• Auto-killing the stale processes — the pre-flight reports and aborts; it does not reach in and kill processes it didn't spawn.

Related: parent/sibling #52 (re-assert EV presence on re-arm). This is test-infra hardening, not part of #52's behavior change.

[JakeMG-INL]

This was generated by AI during triage.

Item 3 (per-run transport isolation) split out to #63 (enhancement / needs-triage). The agent brief above is scoped to items 1 + 2 only.

[JakeMG-INL]

Implementation summary (agent, 2026-06-16)

Branch / base: development @ 6513664
Files touched:

• tests/conformance/e2e/conftest.py (new) — fail-fast pre-flight + per-test timeout backstop
• tests/conformance/e2e/test_stale_emulator_guard.py (new) — regression tests
• requirements.txt — add pytest-timeout==2.4.0
• tests/conformance/README.md — operator note on the pre-flight
• docs/adr/0003-conformance-test-framework.md — add pytest-timeout to the test-dep enumeration

Tests added:

• test_matcher_flags_launched_emulator / test_matcher_ignores_non_emulator_cmdlines — cmdline matcher
• test_scan_detects_a_live_emulator_named_process — real live-process detection by PID
• test_e2e_items_carry_the_wall_clock_backstop — collection hook attaches the timeout cap
• test_message_names_every_offending_pid — diagnostic names PIDs + recovery hint

Verify: PASS

• Stale emulator alive → E2E aborts in 0.35s naming PID 58924, exit code 1 (AC1).
• Clean run unaffected: 19 passed in 60.10s for the E2E layer; full conformance suite 332 passed, 5 skipped in 60.36s (AC2).
• pytest-timeout enforcement confirmed via a 1s probe: aborts with traceback Failed: Timeout (>1.0s) (AC3 mechanism).

Decisions made:

• Detection via psutil, not pgrep — psutil is already a project dep; gives a structured cmdline (exact basename match), no external-tool dependency, and is unit-testable. The matcher requires a Python interpreter as argv0 so an editor/grep that merely names the script isn't flagged.
• Guard lives in a new tests/conformance/e2e/conftest.py — the issue named this path; it didn't exist (shared fixtures are in the parent conftest.py). Putting it here scopes the guard to the E2E layer so codec/state-machine-only runs are unaffected.
• pytest.exit(returncode=1) for a single emphatic fail-fast message (vs N identical per-test errors). returncode is required — a bare pytest.exit() reports success and would let CI go green.
• Backstop = 180s, not the issue's 30-60s — the layer's own in-test deadlines already reach 120s (ISO-20 scenarios / test_iso20_envelope), plus a ~15s SECC-ready wait and ~10s teardown ≈ 145s worst-case legitimate single test. A 30-60s cap would flake legitimate slow runs, notably the loaded AcCCS-box Pi gate, violating AC2. 180s bounds a true hang without false kills. The fail-fast pre-flight (item 1) is what actually catches the Fail fast in conformance E2E when stale emulator processes are alive (cross-talk hang) #62 scenario in sub-second; the timeout is the defense-in-depth backstop item 2 asks for.
• Timeout applied via pytest_collection_modifyitems scoped to the e2e dir (covers future E2E tests with no per-file edit; leaves the fast layers' tight speed targets alone), guarded on plugin presence so it's inert if pytest-timeout is missing.

Open caveats:

• The full 180s abort wasn't demonstrated end-to-end (impractical); proved the pytest-timeout mechanism with a 1s probe and verified the 180s marker is attached to E2E items.
• Optional item Open-source contributions from Ford #3 (per-run transport isolation via netns/unique veth/multicast group) deliberately not done — the issue says to split it into its own issue if pursued.

[JakeMG-INL]

Follow-on from /audit-issue: matcher-precision refinement split out to #64 (enhancement / needs-triage) — tighten _cmdline_runs_emulator to the script position. Non-blocking; #62's fail-fast pre-flight ships as audited.

[JakeMG-INL]

Landed in baf15d2 on development.

Fail-fast pre-flight (psutil process scan) aborts the conformance E2E in sub-second with the offending PIDs when a stale run_secc.py/run_evcc.py is alive, plus a 180s pytest-timeout backstop on every E2E test. Scoped to the e2e layer; codec/state-machine runs unaffected.

Manual verification: independently reproduced AC1 (stale emulator → 0.32s abort naming PID, exit 1), AC2 (clean run 332 passed / 5 skipped in ~60s), and the pytest-timeout enforcement, plus confirmed the 180s marker attaches to all six e2e files.

Follow-on: #64 (tighten the cmdline matcher to the script position).