No required check runs the Python test suite: five of six gate the retired Rust crate

[hkngln]

Scope

Deep review of dev at ac79d4c7ffe00e43f26f3b7c6265436afbff5b0e.

Problem

Branch protection on both main and dev requires the identical six contexts, with strict: true and enforce_admins: true. Measured this session:

$ gh api repos/alplabai/tan-cli/branches/dev/protection --jq .required_status_checks.contexts
["lint","test (ubuntu-latest)","test (windows-latest)","test (macos-latest)","msrv","seam1 -- plan-shape parity"]

$ gh api repos/alplabai/tan-cli/branches/main/protection --jq .required_status_checks.contexts
["lint","test (ubuntu-latest)","test (windows-latest)","test (macos-latest)","msrv","seam1 -- plan-shape parity"]

$ gh api repos/alplabai/tan-cli/branches/dev/protection --jq '{strict: .required_status_checks.strict, enforce_admins: .enforce_admins.enabled}'
{"enforce_admins":true,"strict":true}

Five of the six are cargo jobs on the crate being retired (#269):

• lint — .github/workflows/ci.yml:40, running cargo fmt --all --check (:53) and cargo clippy --all-targets --locked -- -D warnings (:55)
• test (ubuntu-latest) / test (windows-latest) / test (macos-latest) — .github/workflows/ci.yml:59, running cargo build --all-targets --locked (:72) and cargo test --locked (:74)
• msrv — .github/workflows/ci.yml:190, running cargo check --workspace --all-targets --locked (:209)

The sixth, seam1 -- plan-shape parity (.github/workflows/parity.yml:218-219), is the only required context that runs any Python. Its Python selection is exactly three invocations:

:361  python3 -m pytest tests/parity/test_seam1_field_diff.py -q            (repo-root tests/)
:370  python3 -m pytest tests/parity/test_seam1_tan_reconciliation.py -q    (repo-root tests/)
:561  python3 -m pytest tests/gates -v --deselect tests/gates/test_jlink_aen_device_freshness.py
:595  python3 -m pytest tests/gates/test_jlink_aen_device_freshness.py -q

The job that actually runs the port's test suite AND builds the oracle it compares against is python-tests in .github/workflows/parity.yml:1273-1274, whose context names are python -- pytest across python/ (ubuntu-latest) / (windows-latest) / (macos-latest). It does cargo build --locked --bin tan (:1321) and then python -m pytest -q --ignore=tests/gates (:1328) with ALP_SDK_ROOT bound. None of its three contexts is required. Neither is python (.github/workflows/ci.yml:87, python -m pytest tests -q at :164).

The size of the hole, collected on this tree:

$ python -m pytest tests -q --collect-only -p no:cacheprovider   # from python/
2716 tests collected in 0.61s

$ python -m pytest tests/gates -q --collect-only -p no:cacheprovider
347 tests collected in 0.15s

tests/gates — 347 tests — is the entire python/tests slice any required context executes. The remaining 2369 of 2716 collected tests (87%) have no vote on a merge, including tests/commands (1406), tests/core (519), tests/parity (346), tests/conformance (25) and tests/installers (15). By line count that is 41,539 of the 44,857 lines under python/tests.

Separately, ci.yml's python job has no cargo build step at all, so the 18 test functions decorated with an oracle-presence skipif — for example python/tests/parity/test_support_bundle_oracle_parity.py:65-68:

_ORACLE_REQUIRED = pytest.mark.skipif(
    RUST is None,
    reason="needs a built Rust tan; run `cargo build --bin tan` (or set TAN_RUST_BINARY)",
)

— skip in that job across all five files that define it (test_run_oracle_parity.py, test_support_bundle_oracle_parity.py, test_build_sdk_root_oracle_parity.py, tests/commands/test_diff_command.py, tests/commands/test_pinmux_command.py). A skip is green, so even the non-required python context reports success having never compared the two binaries.

Impact

The python -- pytest across python/ legs and python do run on every PR and did pass on the last merge (#352, merged into dev), so nothing is broken today. The defect is that nothing stops them from going red and the PR merging anyway: with all six required contexts green, GitHub permits the merge, and enforce_admins: true gives no extra protection because the failing job is not in the list to begin with.

Concretely, a regression in python/tan/envelope.py or any command's --format json output turns python/tests/commands red (1406 tests, none required) while lint, the three cargo test legs, msrv and seam1 -- plan-shape parity all stay green — the six checks that gate the merge compile a different language in a different directory and cannot see the change. The {command,ok,exitCode,project,data,issues} envelope is the contract alp-sdk-vscode consumes, so the merge that breaks the extension's parsing is exactly the merge no required check can block. The same holds for python/tests/commands/test_flash_command.py (142 tests), which covers the refusals standing between a user and a wrong-device flash.

The exposure grows, not shrinks: #269 deletes crates/, which removes the five cargo contexts' subject matter entirely and would leave seam1 -- plan-shape parity — 347 gate tests — as the sole gate on the whole repository.

Acceptance criteria

• Add python -- pytest across python/ (ubuntu-latest), python -- pytest across python/ (windows-latest) and python -- pytest across python/ (macos-latest) to required_status_checks.contexts on both main and dev; verify with gh api repos/alplabai/tan-cli/branches/{dev,main}/protection --jq .required_status_checks.contexts.
• Add the python context from .github/workflows/ci.yml:87 as well, or state in that job's comment why it is deliberately advisory.
• Add a cargo build --locked --bin tan step to ci.yml's python job, so the 18 _ORACLE_REQUIRED tests execute there instead of skipping green — or make that job assert that none of them skipped, the way .github/workflows/parity.yml:561-582 already asserts PASSED by node id for the two planner-freshness tests.
• Add a test under python/tests/gates/ — the only slice a required context runs today — that reads .github/workflows/*.yml, collects every job whose steps invoke pytest against python/tests, and fails when any of the resulting context names is absent from a committed list of required contexts. That test would have caught this, and it keeps catching it when a new suite is added under a new job name.
• Once crates/ is removed (Retire the Rust oracle: delete crates/, and decide who enforces contract/ afterwards #269), demote lint, the three test (<os>) legs and msrv out of the required set in the same change that deletes the jobs, so protection is never left requiring contexts that no workflow can report.

Related: #269.

[alpCaner]

Branch protection updated — the repo-setting half is done

Applied to both main and dev, verified by reading the API back rather than trusting the write:

strict=true  enforce_admins=true
contexts:
  lint
  test (ubuntu-latest)
  test (windows-latest)
  test (macos-latest)
  msrv
  seam1 -- plan-shape parity
  python -- pytest across python/ (ubuntu-latest)      <- added
  python -- pytest across python/ (windows-latest)     <- added
  python -- pytest across python/ (macos-latest)       <- added

Six contexts -> nine. A Python regression on any of the three OSes now blocks a merge to either branch.

All three OSes rather than ubuntu-latest alone, deliberately: tan-cli#297 and #306, and the alp-sdk-vscode archive failures, were each single-platform. A ubuntu-only requirement would have caught none of them, and the cargo test (*) matrix already requires all three.

The jobs themselves were already running and already green on ac79d4c — this issue was never about missing CI, only about which results could be merged past.

Worth recording: the first attempt failed on both branches

$ gh api -X PATCH .../required_status_checks -f strict=true ...
{"message":"Invalid request.\n\nFor 'properties/strict', \"true\" is not a boolean.","status":"422"}

gh api -f sends every value as a string. The API rejected the whole request rather than applying the contexts and dropping strict, which is the good failure mode — a partial apply would have silently weakened protection while looking like it worked. Reapplied with a JSON body.

Backups of the pre-change rulesets for both branches were taken first and are held outside the repo.

Not changed, and why

msrv and the three cargo test (*) contexts stay required. They gate the frozen crates/ oracle (#269), so there is a real argument for dropping them — but that is a separate decision and not one to take while cutting a release.

Still open on this issue

The repo-side half: confirming a stable job exists for the full shipping suite, what seam1 -- plan-shape parity actually executes today, and whether tests/parity/scaffold_byte_parity.py runs anywhere with an SDK bound. That last one matters — measured on this tree, it skips silently with no SDK and fails, exit 1, five trees with ALP_SDK_ROOT bound, so it currently passes CI by not running.

[hkngln]

Resolved — verified by measurement, not by this change.

Both branches now require all three Python legs. Measured just now:

$ gh api repos/alplabai/tan-cli/branches/dev/protection --jq '.required_status_checks.contexts'
["lint","test (ubuntu-latest)","test (windows-latest)","test (macos-latest)","msrv",
 "seam1 -- plan-shape parity",
 "python -- pytest across python/ (ubuntu-latest)",
 "python -- pytest across python/ (windows-latest)",
 "python -- pytest across python/ (macos-latest)"]

main returns the identical list. Nine contexts, not the six this issue was filed against, and python -- pytest across python/ is the job that both builds the oracle (cargo build --locked --bin tan) and runs the whole suite — so the ~2369 of 2716 collected tests that had no vote now have one, on all three OSes.

Two follow-ups that are deliberately NOT folded in here:

1. python (.github/workflows/ci.yml:87) is still not required — measured 0 on both branches. That is the pip-install-the-declared-package gate, which exists because a release once froze code no CI had imported. Worth adding; it already runs on every PR, so requiring it is instant and reversible. Left as a settings decision for the maintainer rather than taken unilaterally.
2. Ordering against Retire the Rust oracle: delete crates/, and decide who enforces contract/ afterwards #269. The five cargo contexts must be removed from the required list before the PR that deletes crates/ merges, not after. On pull_request events the workflow files come from the PR's merge ref, so a PR that deletes those jobs while they are still required blocks forever on "Expected — waiting for status" and needs an admin merge.

Also worth carrying into #269: seam2 -- materialise + real build + Renode smoke builds and drives the Rust tan (.github/workflows/parity.yml:819-820,844). Deleting crates/ without repointing seam2 at the Python entrypoint removes the only end-to-end build-and-boot gate in the repo.