RELEASE-3 W3: test exact built artifacts before publishing the same files (G-C) - #48
Open
AminDhouib wants to merge 4 commits into
Open
AminDhouib wants to merge 4 commits into
AminDhouib wants to merge 4 commits into
Conversation
…l smoke
plan_RELEASE §2.3 (gap G-C), part 1 of 2: the tools the gate jobs call. No
src/ production edits.
tools/package_verify.py — every packaging fact lives here, so build-dist,
package-verify, install-smoke and publish.yml all ask the SAME code rather
than four shell blocks that can drift:
manifest hash + describe dist/ once, right after the single build
verify re-check a dist/ against that manifest (hashes, sizes,
name/version agreement across manifest + both filenames +
wheel METADATA + sdist PKG-INFO, wheel/sdist membership,
both console entry points, package data)
hash-check the cheap one-file precondition a smoke cell / publish runs
assert-version manifest version == the release tag
Stdlib only, so the jobs that use it need no dependency sync.
embedded/js/*.js are real package data: the cloner engine reads them at
runtime, so a wheel that omits them installs and imports fine and only fails
on first use. verify proves the wheel and the sdist carry byte-identical
copies — comparing the two artifacts to EACH OTHER, not to the git checkout,
so no runner's line-ending policy can make the check lie.
tools/install_smoke.py — installs one artifact into a fresh venv by absolute
path with caches disabled, proves what landed IS the artifact (version, the
package imports from that venv's site-packages, every embedded/js file
matches the hash recorded from the wheel), resolves that env's launcher via
W1's resolve_launcher, and runs W1's canonical journey UNCHANGED. There is
one journey and this is not a second one.
tools/corrupt_artifact.py — the bite-proof fixture. Copy-only by
construction: it refuses to write to its source and re-checks the source
size afterwards, so a negative test can never damage the run's real hashed
artifact.
tests/test_package_verify.py — 23 hermetic pins over synthetic wheels/sdists
(no network, no uv build). Every rule is proven to BITE, not merely to pass:
flipped byte, dropped js member (rejected at manifest time AND by verify),
wheel/sdist package-data disagreement, missing module, missing entry point,
metadata/filename version disagreement, stray .js outside the package,
sdist missing package data, manifest schema version, and the tag
precondition.
.gitignore — found while inspecting the real sdist: .claude/worktrees/ holds
a FULL checkout per agent worktree, and hatchling swept all of it into the
distribution. A local uv build produced a 56 MB sdist of which 55 MB was
that directory; ignoring it gives 14 MB. It also stops ruff linting three
extra copies of the source tree (501 files scanned -> 129).
Local: ruff format/check, ty (76 baseline), vulture, budgets, suppression
owners all clean; unit lane 784 passed (761 pre-existing + 23 new).
Both artifact kinds additionally smoked end-to-end on Windows against the
real uv build output, and the three bite proofs rehearsed against it.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
… files
plan_RELEASE §2.3 (gap G-C), part 2 of 2: the topology. Job semantics stay in
the ONE reusable release-gate.yml; publish.yml CALLS it rather than
re-implementing a second source of truth.
release-gate.yml gains three edges, all wired directly into the stable
`release-gate` aggregate (which already fails on any non-success dependency,
including skipped/cancelled/missing):
build-dist the ONLY `uv build` in the run. Validates metadata and
package data, hashes everything into release-manifest.json,
and uploads dist/ as the run's one immutable artifact.
Exposes the built version as a workflow output.
package-verify downloads that artifact and re-checks it independently,
then runs three IN-JOB BITE PROOFS against throwaway
copies (the real artifact is never touched, and no branch,
commit, push, tag or publication is involved):
1/3 a flipped byte is rejected by the hash check
2/3 a wheel missing embedded/js is rejected — the copy is
re-hashed into its OWN manifest first, so only the
membership rule can reject it, independent of hashing
3/3 the publish tag precondition rejects a wrong tag
and finally re-verifies that the real artifact is intact.
install-smoke {wheel, sdist} x {Ubuntu/X64, Windows/X64, macOS/ARM64}:
each cell downloads the artifact, hash-checks it, installs
it into a fresh venv by absolute path with caches disabled,
and runs W1's canonical journey through the launcher that
install produced. No cell rebuilds or fetches from PyPI.
A new `release_tag` input makes the gate itself fail when a tag disagrees
with the built artifact's metadata version, so the disagreement is caught
before any publish job exists to be blocked.
publish.yml is rewritten around the same gate: at the tag SHA it calls
release-gate.yml with `ref` and `release_tag`, so a tag is qualified by
exactly the jobs a pull request runs — including the six install-smoke
cells. `publish` then needs that gate, downloads the already-hashed dist/
from the same run, re-verifies hashes/membership/tag agreement, and uploads
those very files to PyPI and the GitHub release. It never runs `uv build`,
never rebuilds from the checkout, and never re-downloads the version from
PyPI. Any failed, skipped or cancelled cell leaves needs.gate non-success
and nothing is published. The old three-job "unit tests on ubuntu then build
and publish" path is gone; permissions are also narrowed to the publish job.
actionlint 1.7.7 clean on all three workflows (release-gate.yml,
publish.yml, test.yml). The `runner` context is used only at step level,
never in job-level env.
The macOS install-smoke cells are included here deliberately and NOT
pre-excluded: whether F-773 reaches this lane is an empirical question and
this run is the experiment.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
CI round 1 (run 30161880495) answered the empirical question the previous commit posed. Result: 4 of 6 install-smoke cells green (wheel+sdist x Linux/X64, Windows/X64); both macOS/ARM64 cells failed with F-773's exact signature -- about:blank ok in 1.1s/3.1s, a connection to the CLOSED port 127.0.0.1:1 hung 35.3s, the fixture server served NOTHING, the network.mojom.NetworkService process alive throughout, workspace already under RUNNER_TEMP. It is F-773, not a new defect. New information worth having: this reproduced from a FRESH INSTALL of the built wheel and sdist into a brand-new venv, not from the editable checkout. That eliminates install layout / editable install as a factor -- the hang happens through the exact files a user downloads. Recorded in the finding (now 13/13 reproductions) along with what the gate does and does not claim. Rather than drop the cells, they now run a DECLARED PARTIAL stage: release_gate_harness.run_release_gate_journey gains `stages`, selecting how far the ONE journey runs -- never which journey. "full" (default) is unchanged and is what W1's transport test and all four non-macOS smoke cells run. "handshake" stops after the non-navigating prefix: initialize -> tools/list (94) -> list_instances -> the representative parity call. So macOS/ARM64 now proves the published artifacts INSTALL AND SERVE, and proves nothing about navigation. That is a strictly smaller claim, and it is labelled everywhere it could be misread: - the cell name carries "NO-NAVIGATION partial" in the check list - a per-cell warning annotation fires before the step runs - install_smoke prints "PARTIAL ... must not be reported as full-journey" - the result record carries stages + navigation_verified: false - the gap-declaration job enumerates full vs partial coverage explicitly This is deliberately NOT an xfail and NOT continue-on-error: nothing failing is being marked expected-to-fail, and no cell is silently skipped while the gate reports green. A smaller thing is run, and it is named. transport-known-gaps -> known-gaps: the same root cause now bounds a second lane, and gap declarations get ONE home rather than two. Aggregate edge renamed with it. Local: transport marker still passes (the full journey is unchanged), unit lane 784 passed, ruff/ty/vulture/budgets/suppression-owners clean, actionlint clean. Handshake mode exercised end-to-end on Windows. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
…ress
The W3 properties most likely to rot are the ones nothing executes on a
normal PR. A tag build happens rarely and cannot be dry-run from here, so
"publish never rebuilds" and "the aggregate never forgets an edge" were only
review promises. tests/test_release_workflows.py turns them into 14
structural pins over the workflow YAML.
Build once / publish the same files:
- `uv build` (or any build command) appears in build-dist and NOWHERE else
in the gate
- publish.yml never builds and never re-installs the package from PyPI
- publish.yml CALLS the reusable gate with both `ref` and `release_tag`
instead of duplicating job semantics
- the publish job needs that gate, downloads exactly the one `dist`
artifact, and re-verifies it before uploading
Aggregate integrity:
- release-gate's `needs` equals every other job in the file — a lane that
exists but is not an edge can fail while the required check stays green.
This is what will tell W5 to wire `release-evidence` rather than letting
it discover the omission later.
- every listed edge's `.result` is actually asserted in the script
- the aggregate keeps `if: always()` (a failed dependency must FAIL it, not
skip it — a skipped required check does not block merge)
- no job or step anywhere in the gate sets continue-on-error
Declared gap stays declared:
- install-smoke covers wheel AND sdist on all three cells
- macOS/ARM64 is `handshake`, Linux/Windows are `full`
- the partial cells carry "NO-NAVIGATION partial" in the check name
- known-gaps exists, is a required edge, and names both bounded lanes
- the PR caller has no path filter that could omit packaging changes
Each pin was proven to BITE by temporarily mutating the workflow and
watching the specific test go red (dropped aggregate edge -> failure;
macOS flipped to full -> failure; `uv build` added to publish -> failure),
then restoring from git.
pyyaml moves from a transitive dependency to a declared TEST extra so the
pin cannot vanish if uvicorn[standard] drops it. Test-only; the lockfile
change is 2 lines and no version churn.
Local: 798 passed (784 + 14), ruff/ty/vulture/budgets/suppression-owners
and actionlint all clean.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
RELEASE-3 (W3) — gap G-C: build once, test those exact files, publish those exact files
Stacked on
audit/release-fix-c(95ddb75). Zerosrc/production edits.Topology
The distribution is built exactly once per run, in
build-dist, which hashes itinto
release-manifest.jsonand uploadsdist/as the run's one immutable artifact.Everything downstream downloads that artifact:
build-distuv build; metadata + package data valid; SHA-256 + version recordedpackage-verifyinstall-smoke{wheel, sdist}× cells: fresh venv, local install by absolute path with caches disabled, then W1's canonical journey, unchangedpublish.ymlno longer has its own test/build path: at the tag SHA it calls the samereusable gate with
ref+release_tag, then downloads that run's already-hasheddist/, re-verifies it, and uploads those very files. It never runsuv build, neverrebuilds from the checkout, and never re-downloads the version from PyPI. A failed,
skipped, or cancelled cell leaves
needs.gatenon-success and nothing is published.All three new edges are direct
needsof the stablerelease-gateaggregate.One home, not a second mechanism
Every packaging fact lives in
tools/package_verify.py(stdlib only) —manifest,verify,hash-check,assert-version.build-dist,package-verify,install-smoke, andpublish.ymlall call that same code instead of four shell blocksthat can drift.
tools/install_smoke.pyimports W1's journey fromtests/release_gate_harness.pyunchanged; there is no second journey and no secondfixture mechanism.
embedded/jsis real package dataThe cloner engine reads those seven scripts at runtime, so a wheel that omits them
installs and imports fine and only fails on first use.
verifyproves the wheel and thesdist carry byte-identical copies (comparing the two artifacts to each other, not
to the git checkout, so no runner's line-ending policy can make the check lie), and the
smoke proves the files landing in a fresh
site-packagesare those same bytes.Bite proofs — no repository mutation
Run in
package-verifyagainst throwaway copies.tools/corrupt_artifact.pyiscopy-only by construction: it refuses to write to its source and re-checks the source
size afterwards. No temp branch, commit, push, tag, or publication.
hash-checkrejects it.extract_styles.js→ the copy is re-hashed into its own manifestfirst, so only the membership rule can reject it, independent of hashing. Both
manifestandverifyreject.assert-version(the publish precondition) rejects it.Then the job re-verifies that the real artifact is still intact. The same rules are
pinned hermetically in
tests/test_package_verify.py(23 tests, synthetic wheels/sdists,no network) so each one is proven to bite rather than merely to pass.
macOS install-smoke and F-773 - the experiment, and its answer
The six cells were pushed unexcluded in the first round, because whether F-773
reaches this lane is an empirical question, not one to assume. It does.
Round 1 (run
30161880495): 4 of 6 green - wheel+sdist on Linux/X64 andWindows/X64. Both macOS/ARM64 cells failed with F-773's exact signature:
about:blankok in 1.1s/3.1s, a connection to the closed port127.0.0.1:1hung 35.3s, the fixture server served nothing, the
network.mojom.NetworkServiceprocess alive throughout, workspace already under
RUNNER_TEMP.New information: this reproduced from a fresh install of the built wheel and
sdist, not from the editable checkout - so install layout is eliminated as a
factor. The hang happens through the exact files a user downloads. Recorded in
finding_F773_macos_detached_navigation.md(now 13/13 reproductions).The macOS cells therefore now run a declared partial stage: install + launcher
resolve +
initialize+tools/list(94) +list_instances, and nonavigation.
run_release_gate_journeygained astagesparameter selecting howfar the ONE journey runs - never which journey. This is not an xfail and
not
continue-on-error: nothing failing is marked expected-to-fail, and nocell is silently skipped. A strictly smaller claim is made, and it is named in the
check name (
NO-NAVIGATION partial), in a per-cell warning annotation, in thetool's stdout, in the result record (
stages,navigation_verified: false), andin the
known-gapsjob (renamed fromtransport-known-gaps- one home for gapdeclarations, now that the same root cause bounds two lanes).
So the gate claims, on macOS/ARM64: the published artifacts install and serve.
It does not claim they navigate.
Topology pins
tests/test_release_workflows.py(14 tests) turns the properties nothing executeson a normal PR into assertions: publish never builds, publish calls the reusable
gate and re-verifies what it downloads, the aggregate's
needsequals every otherjob in the file, every edge's result is actually checked,
if: always()is intact,nothing sets
continue-on-error, and the macOS cells stayhandshakewhile theothers stay
full. Each was proven to bite by temporarily mutating the workflowand watching the specific test go red.
Observed flake (routed, NOT fixed)
Chrome cold start intermittently fails outright on hosted Ubuntu runners with
nodriver's
Failed to connect to browser, after the harness's two warmup attemptsare exhausted. Seen 3 times across 3 runs (
transport (Linux/X64)once;both
install-smoke (*/Linux/X64)cells once), each time passing on a plainre-run with identical code and identical Chrome (150.0.7871.128). It is not
specific to W3 - it hit W1's
transportlane too - and it is environmental, not apackaging defect. Flagged to the team lead for an F-id; deliberately not fixed
here, since the remedy lives in the harness's warmup/retry policy which W3 does
not own.
Incidental find
.claude/worktrees/(a full repo checkout per agent worktree) was untracked andun-ignored, so hatchling swept it into the sdist: a local
uv buildproduced a 56 MBsdist of which 55 MB was that directory. Now gitignored → 14 MB. It also stops ruff
linting three extra copies of the source tree (501 files scanned → 129).
Local verification
ruff format/check,ty(76-diagnostic baseline),vulture, file budgets, andsuppression owners all clean. Unit lane 784 passed (761 pre-existing + 23 new).
actionlint1.7.7 clean on all three workflows. Both artifact kinds were additionallysmoked end-to-end on Windows against the real
uv buildoutput, and all three biteproofs rehearsed against it, before the first CI round.