Skip to content

fix(ci): the publish-verification bound failed a release that had shipped - #16

Merged
hseshadr merged 2 commits into
mainfrom
fix/publish-verify-propagation-bound
Aug 3, 2026
Merged

fix(ci): the publish-verification bound failed a release that had shipped#16
hseshadr merged 2 commits into
mainfrom
fix/publish-verify-propagation-bound

Conversation

@hseshadr

@hseshadr hseshadr commented Aug 3, 2026

Copy link
Copy Markdown
Owner

The claim this touches

A green publish run means the registry actually serves the version.

That claim was false in both directions. The ci bricks already fixed the
false-positive half — six green publish runs once stacked up on a package that 404'd on
PyPI, so the verify step asks the registry instead of trusting the uploader. This PR fixes
the false-negative half: the bound was so tight it failed releases that had genuinely
succeeded.

The design is unchanged and stays. Ask the registry, never trust the uploader, and a
timeout is a FAILURE — never a pass, never a warning, no continue-on-error.

What was wrong

The bound was 6 attempts 10s apart (~60s). Measured 2026-08-03 across four real publishes,
time from a green upload until the registry served the version:

publish served after
PyPI edgeproc-core 0.4.0 >60s — failed this check while genuinely live
PyPI edge-proc 0.3.0 ~120s
npm @edgeproc/errors 0.1.0 (first publish of a new name) ~200s
npm @edgeproc/avow 0.3.0 ~60s

edgeproc-core run 30842985605
is red and edgeproc-core 0.4.0 is on PyPI. A release recorded as failed while it
succeeded is not harmless — it teaches the reader to wave off red publish runs, which is
exactly how the six-green-while-404 problem returns.

The new bound, and why that number

14 attempts, backoff 5 / 10 / 15 / 30 then 60s — 600s of sleep, exactly.

  • 600s is 3x the slowest case measured (~200s). The slow case is the first publish of a
    brand-new package name, and that is also the case a genuine trusted-publisher
    misconfiguration is indistinguishable from. So the budget has to clear propagation by a
    wide margin before the step is allowed to accuse.
  • Backoff, not flat. A flat 60s interval would make the common case pay a fixed toll on
    every attempt. With backoff the normal publish still verifies in ~15s, and only a genuine
    problem pays the full ten minutes.
  • No timeout-minutes is set on these jobs (default 360), so 600s of sleep plus worst-case
    curl time fits comfortably.

The failure message now separates the two causes, because they are no longer equally
likely:

(1) STILL PROPAGATING — open the URL yourself; the slowest propagation measured for
this portfolio was ~200s, so 10 minutes should be ample, but if it loads now then the
release DID happen and this is a false alarm worth reporting.
(2) THE RELEASE NEVER HAPPENED — an incomplete trusted-publisher bootstrap...
Do not re-run this job merely to make it green.

The old text listed only the misconfiguration causes, which is misleading now that a
timeout is more often propagation.

Proof the guard still has teeth

A guard that can no longer fail would be worse than the false negative. The harness pulls
the step's committed run: body out of the workflow YAML and executes it verbatim
under GitHub Actions' own shell flags (bash -e -o pipefail), so these exit codes are the
exit codes the job would produce.

case expected exit code
PyPI edgeproc-core==99.99.99 (absent) — python-publish.yml fail after full budget 1
npm @edgeproc/errors@99.99.99 (absent) — ts-publish.yml fail after full budget 1
PyPI edgeproc-core==0.4.0 (served) pass 0, 1s
npm @edgeproc/errors@0.1.0 (served) pass 0, 1s

The positive controls are there on purpose: a harness that can only produce failures proves
nothing about the failures it produces.

examples/ too

examples/edgeproc-core/publish.yml, examples/edge-proc/publish.yml and
examples/assay/publish.yml each carry an inline copy of this step. examples/ is the
surface consumers copy, so leaving the known-bad bound there would ship the defect to the
next adopter. All three get the identical text. No uses: pin was touched.

Blast radius

This reaches nobody yet. Consumers pin hseshadr/ci by full commit SHA, so the change
is inert until a release is cut and each consumer re-pins. No input, output or permission
moved — the re-pin is the only caller-side work.

Gates

check exit
GH_TOKEN=... uvx zizmor@1.26.1 . 0 — "No findings to report"
tests/security-policy.sh 0 — "Security policy checks passed."
tests/lint-examples.sh (actionlint + zizmor over examples/ + fidelity) 0 — 173 references resolved, 0 MISSING, 0 UNVERIFIABLE across 7 consumers

🤖 Generated with Claude Code

https://claude.ai/code/session_015o7tjWLFZvzRv4KyNfDukx

…pped

The check is right and stays: ask the registry whether the version is
served, never trust the uploader, and treat a timeout as a FAILURE. Its
bound was wrong.

6 attempts 10s apart (~60s) against measured propagation:

  PyPI edgeproc-core 0.4.0     >60s   failed while genuinely live
  PyPI edge-proc 0.3.0         ~120s
  npm  @edgeproc/errors 0.1.0  ~200s  first publish of a NEW name
  npm  @edgeproc/avow 0.3.0    ~60s

edgeproc-core run 30842985605 is red and 0.4.0 is on PyPI. A red run on a
live release is not harmless: it teaches the reader to wave off red
publish runs, which is exactly how the six-green-while-404 defect returns.

New bound: 14 attempts with backoff (5, 10, 15, 30, then 60s) = 600s of
sleep, 3x the slowest case measured, while the common case still verifies
in ~15s. The failure message now separates STILL PROPAGATING from THE
RELEASE NEVER HAPPENED; it previously listed only the misconfiguration
causes, which misleads now that a timeout is more often propagation.

The three examples/*/publish.yml inline copies get the identical text, so
the surface consumers copy does not ship the defect. No input, output or
permission moved; consumers must re-pin to get this.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_015o7tjWLFZvzRv4KyNfDukx
@cursor

cursor Bot commented Aug 3, 2026

Copy link
Copy Markdown

Bugbot is not enabled for your account, so this pull request was not reviewed.

Enable Bugbot in the Cursor dashboard to get automatic reviews on future PRs.

The currency guard demands every hseshadr/ci self-reference match the newest
release. Cutting ci-v3.2.0 left all 41 refs at ci-v3.1.0, so the next PR in this
repo fails the guard before it is even read — this one did.

That is the release-commit bootstrap working as designed (a commit cannot contain
its own SHA), but it means a release should be followed immediately by a re-pin
rather than leaving the debt for whoever opens the next PR.

41 refs moved, 0 left at the old SHA — asserted, not assumed.
security-policy.sh exit 0, zizmor exit 0.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
@hseshadr
hseshadr merged commit 605e51c into main Aug 3, 2026
4 checks passed
@hseshadr
hseshadr deleted the fix/publish-verify-propagation-bound branch August 3, 2026 19:17
@hseshadr

hseshadr commented Aug 3, 2026

Copy link
Copy Markdown
Owner Author

The red Security policy check is not from this PR

validate_first_party_release_lineage fails with:

FAIL: first-party ref 33c5e5f… is a superseded release, not ci-v3.2.0 (7226072…) — re-pin

It names 23 files, including many this PR never opens (examples/aml-filter/*,
examples/almamesh/*, .github/workflows/python-gate.yml, …).

Why it fires

The guard has a documented "release-commit bootstrap" exemption: a commit cannot contain
its own SHA, so the tagged release commit is allowed to carry pins from the previous
release. That exemption fires only when HEAD is exactly the newest release tag's
commit
.

ci-v3.2.0 was cut at 7226072, which is current main. So:

  • on main, HEAD == ci-v3.2.0 → exemption applies → green
  • on any commit on top of main, HEAD != ci-v3.2.0 → strict currency applies → red

The last green main run (30822382547, 14:22:30Z) started three seconds before the
ci-v3.2.0 release existed (14:22:33Z), so it never evaluated this.

Control, run just now

An empty commit on origin/main — zero file changes — fails the same check:

checkout tests/security-policy.sh
origin/main (7226072, == ci-v3.2.0) untouched exit 0
git commit --allow-empty on top of it exit 1, identical superseded release message

So the first ordinary commit after any release is red until the 23 self-pins are moved to
that release's SHA. That is a separate change with its own blast radius, it is entangled
with release mechanics, and it is deliberately not bundled into this bound fix.

To unblock: land a pin bump 33c5e5f7226072 (with the # ci-v3.2.0 comment
update) across those 23 files, then rebase this branch — or rebase this branch onto that
bump. Nothing in this PR needs to change.

For the record, tests/security-policy.sh and zizmor were both exit 0 locally on
this diff, because they ran before the commit existed — i.e. at HEAD == ci-v3.2.0, where
the exemption applies. Same code, same result as main.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant