[Conformance Test Framework] Slice 3: AcCCS-box Pi as self-hosted runner for pre-merge E2E

[JakeMG-INL]

What to build

Activate the AcCCS-box Raspberry Pi as a GitHub Actions self-hosted runner so substantial PRs can be label-triggered to execute the conformance suite on real Pi hardware. This catches Pi-specific issues (smbus, PWM PCB I/O, scapy on the Devolo NICs) that veth-only CI cannot, without requiring a full hardware-in-the-loop session against a real EV/EVSE.

ADR-0003 calls this the pre-merge Pi gate. It is not a hard gate on every PR — it's label-triggered (e.g., needs-pi-run) so reviewers can request it when a change is likely to touch Pi-specific paths.

End-to-end behavior delivered:

• A reviewer adds the needs-pi-run label (or comparable trigger) to a PR.
• A CI job runs pytest tests/conformance/ on the AcCCS-box Pi.
• The result reports back on the PR as a check.

In scope:

• Configure the AcCCS-box Pi as a GitHub Actions self-hosted runner (registration, service installation, restart policy).
• Add a GitHub Actions workflow that triggers on the agreed label and runs the conformance suite on the Pi runner.
• Document the operator workflow in tests/conformance/README.md: when to request a Pi run, how to read the result, what to do if the Pi runner is offline.
• Ensure the runner has CAP_NET_ADMIN (or equivalent permissions) to bring up the veth pair.
• Document Pi runner maintenance: rotating the runner token, restarting the service, where logs live.

Out of scope:

• Real hardware-in-the-loop testing against an actual EV or EVSE (that's the major-release acceptance gate, manual).
• Making the Pi run a hard gate on every PR — it stays label-triggered.
• Hardening for multi-PR concurrency on a single Pi (queue / serialize at GitHub's runner level; revisit if it becomes a bottleneck).

Acceptance criteria

• AcCCS-box Pi is registered and active as a self-hosted runner in the GitHub repo.
• A workflow exists that runs the conformance suite on the Pi when the agreed trigger label is applied.
• tests/conformance/README.md documents how a reviewer requests a Pi run and how to interpret the result.
• Pi runner maintenance steps (token rotation, service restart, log location) are documented.
• At least one PR has been used to verify the workflow end-to-end (smoke set green on the Pi).
• Human verification: maintainer has confirmed the Pi runner is stable across at least three runs before closing this issue.

Blocked by

• [Conformance Test Framework] Slice 1: Foundation + walking-skeleton DIN smoke #18 (Conformance Test Framework Slice 1)

[JakeMG-INL]

Implementation summary (agent, 2026-06-04)

Branch / base: development @ a52dadd
Files touched:

• .github/workflows/conformance-pi.yml (new) — label-triggered Pi conformance workflow
• tests/conformance/README.md — added "Requesting a Pi run" + "Pi runner maintenance" sections

Live infra changes (outside the diff):

• Created the needs-pi-run repo label (the trigger).
• Registered the AcCCS-box Pi (pi@10.10.10.10) as self-hosted runner acccs-box — labels ["self-hosted","Linux","ARM64","acccs-box"], installed as systemd service actions.runner.IdahoLabResearch-AcCCS.acccs-box.service, currently online / idle. (Authorized by maintainer this session.)

Tests added: none — CI workflow + docs (chore class). YAML validated via yaml.safe_load.

Verify:

• Workflow YAML parses; runs-on=[self-hosted, acccs-box], if=github.event.label.name == 'needs-pi-run', on=pull_request:[labeled].
• Runner confirmed status: online, busy: false via gh api .../actions/runners.
• End-to-end smoke (criterion Ford merge #5) NOT run — see caveats.

Acceptance criteria status:

• Lack of detail information for PCB #1 Pi registered & active as self-hosted runner — online/idle, correct labels.
• Make AcCCS usable with different hardware setups #2 Label-triggered workflow runs the conformance suite on the Pi.
• Open-source contributions from Ford #3 README documents how to request a Pi run + read the result.
• DIN 70121 fixes (plus EV config prioritization and deprecated datetime replacement) #4 Runner maintenance (token rotation, service restart, log location) documented.
• Ford merge #5 At least one PR smoke-green on the Pi — blocked, see caveats.
• [Personality Config] Slice 1: Loader plumbing, field migration, and default.yaml #6 Maintainer confirms stability across ≥3 runs — human verification, by design.

Decisions made:

• Trigger = needs-pi-run label via on: pull_request: types: [labeled] + job if gate — the issue's own suggested name; standard re-run-by-relabel pattern (documented).
• Runner label = acccs-box (in addition to auto self-hosted/Linux/ARM64) so the workflow targets this specific Pi — registered with --labels acccs-box.
• Pi workflow installs the full requirements.txt (smbus included), unlike the hosted job which strips it — exercising the real smbus/ARM build is the whole point of the Pi gate (ADR-0003).
• Lean workflow (no setup-python, no apt toolchain install): the Pi is a dedicated, pre-provisioned box; prerequisites are documented as operator tasks, not installed per-run.
• Added a security note: never enable this workflow for fork PRs (self-hosted runner executes PR code on real hardware).

Open caveats:

• Blocks Ford merge #5: the runner's service PATH is /usr/local/bin:/usr/bin:/bin:/usr/games. The Pi has Python 3.13 + a venv at /home/pi/AcCCS/venv, but no python3.12, no conda, and no python on the service PATH. So python -m pip install -r requirements.txt will fail until a maintainer exposes a working Python 3.12 (+ deps) on the service PATH — the documented prerequisite. requirements.txt is pinned to 3.12 in hosted CI; whether it installs cleanly on 3.13 is unverified (this is exactly the Pi-specific risk the gate exists to surface). I can provision the Python env on request.
• Ford merge #5 also needs an actual PR to apply the label to — PR creation is /audit-issue's domain, not this skill's.
• Pre-existing, out of scope: README "Gating policy" section still says "three smoke scenarios" while ADR-0003 now lists seven. Not touched.

[JakeMG-INL]

Audit follow-up (2026-06-04)

Audited the unstaged changes (workflow + README) at HEAD a52dadd. The artifacts are correct and verified (runner acccs-box online/idle, needs-pi-run label present, YAML valid, mirrors the in-production conformance.yml). Not shipped — handing back to add a security guard before anything lands, per maintainer choice.

Address before landing

1. Add a technical fork-PR guard — .github/workflows/conformance-pi.yml:46. The job currently gates only on if: github.event.label.name == 'needs-pi-run'. This is a public repo with a self-hosted runner that has passwordless sudo (ip link); a documentation-only "never for forks" invariant is not enough defense-in-depth. Add a head-repo check so the workflow can never execute fork-head code even if a maintainer mis-labels a fork PR:

   if: github.event.label.name == 'needs-pi-run' && github.event.pull_request.head.repo.full_name == github.repository

   Why this blocks landing: without it, the only thing standing between an untrusted fork PR and sudo-on-real-hardware is a human remembering not to label it.

Still-open acceptance criteria (not closeable yet)

These are not code fixes but must be tracked — #6 explicitly forbids closing the issue until satisfied:

• Ford merge #5 — live PR smoke-green on the Pi (blocked). The runner service PATH (/usr/local/bin:/usr/bin:/bin:/usr/games) has no python/python3.12 — only Python 3.13 + a venv at /home/pi/AcCCS/venv. python -m pip install -r requirements.txt will fail until a maintainer exposes the AcCCS 3.12 env (incl. build toolchain + i2c headers) on the service PATH — the prerequisite the README documents. Also needs an actual PR to apply the label to. Whether requirements.txt (pinned 3.12) installs cleanly on 3.13 is unverified — the exact Pi-specific risk this gate exists to surface.
• [Personality Config] Slice 1: Loader plumbing, field migration, and default.yaml #6 — maintainer confirms stability across ≥3 runs. Human verification by design; do not close until done.

Minor (verification fidelity)

• Agent 1 validated the workflow with yaml.safe_load, which coerces the on: key to boolean true (PyYAML "Norway problem") and so never actually confirmed the trigger. The trigger is correct (identical idiom to conformance.yml), but safe_load is not a faithful GHA validator.

[JakeMG-INL]

Pi runner provisioned (2026-06-04)

Provisioned the Pi so the conformance gate can actually run. Acceptance criterion #5's prerequisite is now met; #5 itself still needs the workflow committed + a labeled PR, and #6 is the maintainer 3-run sign-off.

What was discovered: the AcCCS-box Pi is Debian 13 (trixie) / aarch64 with Python 3.13.5 as its only interpreter — no python3.12, and trixie's apt has none. The original "no python on the service PATH" assumption is stale (/usr/bin/python exists). The real blockers were (a) the only python is 3.13, not the 3.12 the docs assume, and (b) system python on trixie is PEP-668 externally-managed, so pip install into it is refused.

Decision (maintainer): run the gate on the box's native Python 3.13, not a conda-installed 3.12. Verified requirements.txt installs cleanly on 3.13 — expy's libcbv2g and smbus==1.1.post2 build from source against the (already present) gcc 14 / cmake 3.31 / ninja / libpython3.13-dev / i2c-tools toolchain; all imports succeed; pytest 9.0.3.

What was done on the Pi:

• Installed the full requirements.txt into /home/pi/AcCCS/venv (Python 3.13).
• Prepended /home/pi/AcCCS/venv/bin to the runner's .path (/home/pi/actions-runner/.path; backup at .path.bak) — the systemd runner does not source bashrc, so the auto-activated venv was invisible to jobs; .path is how jobs get it.
• Restarted the runner service; confirmed the Runner.Listener PATH leads with the venv bin and the runner is online/idle.
• Passwordless sudo for ip link confirmed (the sudo ./setup_veth.sh step will run unattended).

Fold into the fork-guard pass: the workflow + README comments still say "Python 3.12" and "the AcCCS conda env." Those are now inaccurate — correct them to "Python 3.13 venv exposed on the runner .path" alongside the fork-PR if guard, so the audit reviews one coherent diff.

[JakeMG-INL]

Additional workflow/harness requirements (from real-Pi validation, 2026-06-04)

Validating the runner on real hardware surfaced three changes the conformance-pi.yml workflow + harness need before the Pi gate is meaningful. Fold these into the same /implement-issue pass as the fork guard and the 3.12→3.13/conda→venv doc fixes.

1. Drop setup_veth.sh / remove_veth.sh from conformance-pi.yml. The Pi has real acccs_evcc/acccs_secc USB-Ethernet NICs, created and persistently renamed by the operator's ~/rename_eth.sh (MAC-matched systemd .link files; "PEV" → acccs_evcc, "EVSE" → acccs_secc). These are the exact names the conformance harness expects, so E2E runs over them directly. Critically, setup_veth.sh's teardown loop does ip link delete acccs_evcc/acccs_secc — on the Pi it would destroy the real NICs. The Pi prerequisite docs should say "run rename_eth.sh once, ensure both USB adapters are connected and coupled," replacing the veth prerequisite.

2. Run the E2E emulators NON-virtual on the Pi. tests/conformance/conftest.py launch_emulator hard-codes --virtual (line ~127), which stubs the SMBus/I2C relays. That means the Pi gate, as written, never exercises the real relay/PWM path — the whole reason for a hardware gate (ADR-0003). Make --virtual configurable (env var or pytest option), default virtual for hosted CI/dev (no PCB), non-virtual on the Pi. Verified: with --virtual removed, din-happy passes non-virtual on the Pi and actuates the real relays.

3. Add a PKI cert-generation step before pytest (both conformance.yml and conformance-pi.yml). TLS/PnC E2E scenarios need certs; the workflows never generate them (see [Bug] Hosted conformance E2E layer fully red — scenarios never reach SessionStop + CI never generates PKI certs #46). Use create_certs.sh (note: must run with CWD = app/shared/pki, and -v iso-20 is currently broken — [Bug] create_certs.sh -v iso-20 fails to generate the MO/contract chain #48).

Status toward criterion #5

Strong evidence #5 is achievable: the full suite runs 314 passed / 1 flake (#47) / 5 skipped on the real Pi NICs. Remaining blockers are the workflow changes above (+ #44 PyYAML, already in PR #45). #6 (maintainer 3-run sign-off) still by design.

[JakeMG-INL]

Pi workflow runs setup_veth.sh — contradicts the real-NIC substrate (surfaced during #46 audit)

While auditing #46 (hosted-runner CAP_NET_RAW fix), the still-untracked conformance-pi.yml was noted to provision a veth pair on the Pi:

• conformance-pi.yml:68 → sudo ./setup_veth.sh
• conformance-pi.yml:77 → sudo ./remove_veth.sh

This is wrong for the Pi substrate:

• ADR-0003 §Substrate: the Pi runs "real TCP/IPv6, real smbus/PWM/Devolo NIC path" — CI is the veth-only substrate, not the Pi.
• setup_veth.sh creates a veth pair named acccs_secc/acccs_evcc — the same names as the Pi's real USB-Ethernet NICs — so it would collide with / shadow the real interfaces rather than exercise them.

The Pi's conda interpreter already carries cap_net_raw=eip, so it needs no setcap step (that part is fine). The fix here is to drop the veth provisioning steps from the Pi workflow so it runs the E2E layer over the real NICs.

Out of scope for #46 (which only touched the hosted conformance.yml); tracking it here under the Pi-runner slice.

[JakeMG-INL]

Note from /audit-issue 47 (2026-06-05)

While auditing #47 (flaky conformance E2E), the working tree was found to also contain this slice's Pi-runner work, which #47 did not list and must not be bundled into its commit. To isolate it, the #47 flaky-test fix was stashed so the tree now holds only the Pi-runner changes for this issue.

Stashed (the #47 fix — do not touch for this issue):

• git stash@{0} — message: issue #47 flaky-test fix (parked to isolate Pi-runner work for #20)
• Contents: tests/conformance/e2e/test_scenarios.py, tests/conformance/conftest.py, tests/conformance/e2e/test_proc_reader.py
• Recover after [Conformance Test Framework] Slice 3: AcCCS-box Pi as self-hosted runner for pre-merge E2E #20 lands: git stash pop (then re-run /audit-issue 47)

Now in the tree for #20:

• tests/conformance/README.md (modified) — "Requesting a Pi run" + "Pi runner maintenance" sections
• .github/workflows/conformance-pi.yml (new) — label-triggered self-hosted Pi workflow
• .gitignore (modified) — added .claude/ (local agent tooling, should not be committed)

Note: README.md links to conformance-pi.yml, so commit them together.

⚠️ The stash is local to this clone only — it is not pushed anywhere. If this machine is wiped before the stash is popped, the #47 fix is lost (it is still recorded in #47's implementation-summary diff, but not as a recoverable patch).

[JakeMG-INL]

Implementation summary (agent, 2026-06-05)

Branch / base: development @ fd6851a
Files touched:

• .github/workflows/conformance-pi.yml (new) — label-triggered Pi workflow, now with the fork guard, real-NIC substrate, non-virtual run, 3.13/venv docs, and cert step
• .github/workflows/conformance.yml (modified) — added the PKI cert-generation step before pytest
• tests/conformance/conftest.py (modified) — --virtual is now env-configurable (ACCCS_E2E_VIRTUAL)
• tests/conformance/README.md (modified) — Pi sections corrected (3.13/venv, real NICs, non-virtual, fork guard, cert prereq)
• .gitignore (modified) — ignore .claude/ (local agent tooling)

This pass folded in all the accumulated review/real-Pi feedback on the ticket:

1. Fork-PR guard — conformance-pi.yml job if now also requires github.event.pull_request.head.repo.full_name == github.repository, so fork-head code can never run on the self-hosted runner even if mislabeled.
2. Dropped setup_veth.sh/remove_veth.sh from the Pi workflow — those would ip link delete the Pi's identically-named real acccs_evcc/acccs_secc USB NICs. The Pi runs E2E directly over the real NICs.
3. Non-virtual on the Pi — conftest.py launch_emulator reads ACCCS_E2E_VIRTUAL (default = virtual, for hosted CI/dev with no PCB); the Pi workflow sets ACCCS_E2E_VIRTUAL=0 to drop --virtual and drive the real SMBus/I2C relay + PWM path (the point of the Pi gate, ADR-0003 § Substrate and gating).
4. PKI cert-generation step added to both workflows: bash create_certs.sh -v iso-2 with CWD app/shared/pki, so TLS/PnC E2E scenarios have a cert chain ([Bug] Hosted conformance E2E layer fully red — scenarios never reach SessionStop + CI never generates PKI certs #46).
5. Doc corrections — workflow comments + README no longer say "Python 3.12 / AcCCS conda env"; they describe the Python 3.13 venv exposed on the runner .path, the cap_net_raw=eip interpreter cap, and the rename_eth.sh real-NIC prerequisite.

Tests added: none — CI workflow + docs + a small test-harness toggle.

Verify: PASS

• Both workflow YAMLs parse and were validated structurally (not via the bool-coercing safe_load key lookup the earlier audit flagged): pi trigger = pull_request: {types: [labeled]}, job if carries both the label and head-repo checks, runs-on=[self-hosted, acccs-box], cert step working-directory: app/shared/pki, run step env.ACCCS_E2E_VIRTUAL: "0".
• _virtual_mode_enabled() confirmed: unset/1/true/yes → virtual on; 0/false/off/no/empty → off.
• Full conformance E2E layer: 11 passed in default (virtual) mode, including all four TLS scenarios — which only pass because the local cert tree was generated exactly the way the new step does (create_certs.sh -v iso-2), confirming iso15118_2/certs is the one tree every protocol reads (app/shared/security.py:1496).

Decisions made:

• Cert step uses -v iso-2 only — CertPath resolves every protocol (incl. iso-20) under iso15118_2/certs, and -v iso-20 is broken ([Bug] create_certs.sh -v iso-20 fails to generate the MO/contract chain #48). One generation covers all TLS scenarios.
• Env-var name ACCCS_E2E_VIRTUAL, default-on — keeps hosted CI and developer machines (no PCB) on the stubbed path with zero config; only the Pi opts out.
• Kept setup_veth.sh in the hosted conformance.yml (veth is correct there) and only added the cert step to it.

Open caveats:

• Ford merge #5 (live PR smoke-green on the Pi) and [Personality Config] Slice 1: Loader plumbing, field migration, and default.yaml #6 (maintainer ≥3-run sign-off) remain open by design — they need the workflow committed + a labeled in-repo PR, and the maintainer's hardware confirmation. The non-virtual real-relay path can only be exercised on the Pi, not locally.
• Stash-pop conflict for [Flaky] Conformance E2E din-variant intermittently fails (done=False) under full-suite runs #47: the parked stash@{0} ([Flaky] Conformance E2E din-variant intermittently fails (done=False) under full-suite runs #47 flaky-test fix) also edits conftest.py launch_emulator (adds PYTHONUNBUFFERED), in the same region this pass changed (--virtual is now conditional). git stash pop after [Conformance Test Framework] Slice 3: AcCCS-box Pi as self-hosted runner for pre-merge E2E #20 lands will conflict there — resolution is trivial (keep both: the conditional --virtual and the env= addition). Flagging so whoever pops it expects it.
• Pre-existing, untouched: README "Gating policy" still says "three smoke scenarios" while ADR-0003 now lists seven — out of scope for [Conformance Test Framework] Slice 3: AcCCS-box Pi as self-hosted runner for pre-merge E2E #20.