Skip to content

action.yml: capability precheck for a stale version: pin on the anchor/attest inputs #114

Description

@SollanSystems

Problem

action.yml has a version: input: non-empty installs loop-engineer[schemas,yaml]==$LOOP_VERSION from PyPI, empty installs from the action's own checkout. So

- uses: SollanSystems/loop-engineer@v0.12.0
  with:
    version: "0.11.0"
    attest: true          # or: anchor: loop-anchor.json

runs v0.12.0's action.yml steps against a 0.11.0 CLI that has none of the surfaces those steps invoke. The adopter gets a failure that blames the wrong input.

Affected surfaces — exactly four

Verified against action.yml at 6fddb3b. These are the only loop invocations that can break on an old pin:

Invocation Step Gated by input
loop verdict "$LOOP_PATH" verdict predicate (:170) attest
loop verdict --emit-subject "$LOOP_PATH" chain-head subject file (:187) attest
loop doctor --expect-chain-ancestor "$ANCHOR_HEAD" "$LOOP_PATH" compare the attested verdict (:252) anchor
loop verdict --compare "$PREDICATE" "$LOOP_PATH" compare the attested verdict (:261) anchor

Not affected, despite appearances:

  • loop doctor --anchor — the anchor input is routed to scripts/action_anchor_resolve.py, where --anchor is that script's own argparse flag. The script self-bootstraps via sys.path.insert(0, _REPO_ROOT) with _REPO_ROOT derived from __file__ under github.action_path, so it imports the action's kernel, not the pinned install. No action step passes --anchor to loop.
  • signer-workflow — same path, same reason. It never reaches the pinned CLI.
  • The loop doctor hard gate (only --expect-chain-head, present since 0.10.0), chain head (pure Python over doctor.json), loop inspect, and the PR-comment step.

Current behavior: confusing, not silent — and only by accident

loop verdict <ws>                              -> exit 2, "unknown loop command: verdict"
loop doctor --expect-chain-ancestor <hex> <ws> -> exit 2, "target path does not exist: --expect-chain-ancestor"

Neither names the real cause. An adopter debugging the second will chase their path: input.

The load-bearing nuance: the exit 2 is an accident of argument order, not a guard. On ≤0.11.0 there was no unknown-flag guard — the flag name simply became argv[0] and failed the target-exists check. Put the same flag after the path and 0.11.0 exits 0 with the flag silently dropped. Measured against git archive v0.11.0 (tree self-identifies as 0.11.0):

loop doctor <ws> --expect-chain-ancestor 000…0  -> EXIT 0, {"ok": true}
loop doctor <ws> --totally-made-up-flag xyz     -> EXIT 0, {"ok": true}

#113 fixed that for 0.12.0+, but old pins cannot be fixed retroactively. action.yml:252 happens to use the flag-before-path form, so today it fails loudly. Reorder that line — or have a consumer write the after-path form in their own workflow against an old pin — and the tamper gate silently passes.

Rejected approaches — please do not re-walk these

1. _COMMANDS membership probed via python - <<'PY'. Wrong in three independent ways:

  • It measures a different package than the steps it guards. python - (stdin) puts the CWD at sys.path[0]; in Actions the CWD is GITHUB_WORKSPACE, the consumer's repo. Any consumer repo containing a loop/ directory shadows the pinned install. Reproduced live: with a real 0.11.0 install plus a stub workspace loop/__main__.py defining _COMMANDS = ("scaffold","doctor","verdict"), the probe printed a clean pass while the installed CLI was 0.11.0. The action's real calls go through the installed console script, whose sys.path[0] is the bin dir. python3 -I - drops CWD and is immune.
  • It probes the wrong fact. _COMMANDS is a private tuple with no stability contract, and command-name membership implies nothing about --emit-subject, --compare, or --expect-chain-ancestor.
  • It would ship with zero CI coverage. All four in-house uses are uses: ./ with no version: (attest.yml:87; ci.yml:151, 172, 308), so a [ -z "$LOOP_VERSION" ] && exit 0 guard short-circuits every run — and those runs have CWD = this repo, which contains loop/, so defect 1 would make it pass regardless.

2. Probing loop <command> --help. There is no per-subcommand help surface: loop/__main__.py routes any post-command --help through the generic unknown-flag path, producing a message indistinguishable from a plain typo. Top-level loop --help does print the installed version's help text and could be scraped, but that has no more stability contract than the _COMMANDS probe.

Suggested design

All four affected surfaces shipped in one release (0.12.0), so a scalar threshold is both sufficient and more robust than per-flag introspection:

  • A single MINIMUM_VERSION = "0.12.0" constant compared against the installed version, parsed as a strict X.Y.Z triple. Carry a comment naming the release and instructing the next surface-adding PR to bump it in the same commit.
  • One new step, immediately after Install loop-engineer and before loop doctor (hard gate).
  • Its "this surface will be needed" decision computed purely from the input strings — never from doctor.json, chain-head, or anchor-outcome, none of which exist yet at that point in the job.
  • The anchor trigger is exactly anchor != '' && expect-chain-head == '' — the same ADR 0002 decision 5 precedence action.yml already implements for the resolve step, so the precheck does not false-fire on an anchor that is set but inert.
  • Emit a composite-action output, e.g. version-precheck-outcome (ok | stale | skipped), mirroring the existing anchor-outcome / chain-head precedent, so CI and consumers get a stable assertion point instead of grepping stderr.

Acceptance criteria

  • The probe resolves the same package the gated steps resolve — the console script, or python3 -I. A test proves a loop/ directory in the workspace does not change the result.
  • Version comparison against a single MINIMUM_VERSION constant, not per-flag or per-command introspection.
  • It cannot silently pass: an absent/unparseable version, an import failure, or an unreadable metadata record fails closed with a message naming version: as the cause.
  • Correct when version: is empty (install-from-checkout) — that path is not a stale pin and must not be blocked.
  • Trigger condition uses the decision-5 precedence above, not anchor != '' alone.
  • Negative-control CI leg. uses: ./ with version: "0.11.0" and attest: true against examples/coverage-repair, asserting the step fails and version-precheck-outcome == 'stale'. Mirror the anchor-live job's continue-on-error + outputs.outcome idiom. A second leg repeats with anchor: "placeholder.json" (expect-chain-head empty) for the anchor branch. No chained event store, github-token, or id-token/attestations permissions are needed — the precheck fires before any of that matters. Per this repo's standard, without a control that fails on the unfixed base this lands untested by our own gates.
  • Tests at two layers, because the gates job never pip-installs loop-engineer: (1) a pure unit test of the version-comparison and input-trigger logic, no subprocess; (2) an integration test building a throwaway bin/ with a loop shim printing a controlled --version, prepended to PATH, with CWD set to a workspace containing a stub loop/__main__.py, asserting the precheck reports the shim's version — mirroring scripts/test_action_anchor_resolve.py's fake-gh-on-PATH pattern.
  • Consider additionally documenting a minimum version on the anchor and attest input descriptions — cheap, and it helps before the probe ever runs.

Notes

Severity is diagnostics, not correctness, for the action as currently written — the job does fail. It is filed because the failure blames the wrong input, and because the silent-pass variant is one line-reordering away for anyone still on a pre-0.12.0 pin.

Out of scope, named rather than left silent: a version floor cannot catch the inverse failure — version: pinned newer than this action.yml's own ref, where that newer release removed or renamed a surface these steps still invoke by name. Catching that needs either a documented append-only CLI-grammar guarantee or a ceiling check. Acknowledged gap, not solved here.

Context: #113 (the generic unknown-flag guard, which fixed the trailing-flag silent pass for 0.12.0+), ADR 0002 decision 5 (the anchor / expect-chain-head precedence).

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions