Skip to content

RELEASE-3 W3: test exact built artifacts before publishing the same files (G-C) - #48

Open
AminDhouib wants to merge 4 commits into
audit/release-fix-cfrom
audit/release-3-w3
Open

AminDhouib wants to merge 4 commits into
audit/release-fix-cfrom
audit/release-3-w3

Conversation

@AminDhouib

@AminDhouib AminDhouib commented Jul 25, 2026

Copy link
Copy Markdown
Member

RELEASE-3 (W3) — gap G-C: build once, test those exact files, publish those exact files

Stacked on audit/release-fix-c (95ddb75). Zero src/ production edits.

Topology

The distribution is built exactly once per run, in build-dist, which hashes it
into release-manifest.json and uploads dist/ as the run's one immutable artifact.
Everything downstream downloads that artifact:

job what it proves
build-dist one uv build; metadata + package data valid; SHA-256 + version recorded
package-verify the downloaded bytes are the built bytes, and the artifacts are structurally complete — plus three in-job bite proofs
install-smoke {wheel, sdist} × cells: fresh venv, local install by absolute path with caches disabled, then W1's canonical journey, unchanged

publish.yml no longer has its own test/build path: at the tag SHA it calls the same
reusable gate
with ref + release_tag, then downloads that run's already-hashed
dist/, re-verifies it, and uploads those very files. It never runs uv build, never
rebuilds from the checkout, and never re-downloads the version from PyPI. A failed,
skipped, or cancelled cell leaves needs.gate non-success and nothing is published.

All three new edges are direct needs of the stable release-gate aggregate.

One home, not a second mechanism

Every packaging fact lives in tools/package_verify.py (stdlib only) — manifest,
verify, hash-check, assert-version. build-dist, package-verify,
install-smoke, and publish.yml all call that same code instead of four shell blocks
that can drift. tools/install_smoke.py imports W1's journey from
tests/release_gate_harness.py unchanged; there is no second journey and no second
fixture mechanism.

embedded/js is real package data

The cloner engine reads those seven scripts at runtime, so a wheel that omits them
installs and imports fine and only fails on first use. verify proves the wheel and the
sdist carry byte-identical copies (comparing the two artifacts to each other, not
to the git checkout, so no runner's line-ending policy can make the check lie), and the
smoke proves the files landing in a fresh site-packages are those same bytes.

Bite proofs — no repository mutation

Run in package-verify against throwaway copies. tools/corrupt_artifact.py is
copy-only by construction: it refuses to write to its source and re-checks the source
size afterwards. No temp branch, commit, push, tag, or publication.

  1. flipped bytehash-check rejects it.
  2. wheel missing extract_styles.js → the copy is re-hashed into its own manifest
    first, so only the membership rule can reject it, independent of hashing. Both
    manifest and verify reject.
  3. wrong tagassert-version (the publish precondition) rejects it.

Then the job re-verifies that the real artifact is still intact. The same rules are
pinned hermetically in tests/test_package_verify.py (23 tests, synthetic wheels/sdists,
no network) so each one is proven to bite rather than merely to pass.

macOS install-smoke and F-773 - the experiment, and its answer

The six cells were pushed unexcluded in the first round, because whether F-773
reaches this lane is an empirical question, not one to assume. It does.

Round 1 (run 30161880495): 4 of 6 green - wheel+sdist on Linux/X64 and
Windows/X64. Both macOS/ARM64 cells failed with F-773's exact signature:
about:blank ok in 1.1s/3.1s, a connection to the closed port 127.0.0.1:1
hung 35.3s, the fixture server served nothing, the network.mojom.NetworkService
process alive throughout, workspace already under RUNNER_TEMP.

New information: this reproduced from a fresh install of the built wheel and
sdist
, not from the editable checkout - so install layout is eliminated as a
factor. The hang happens through the exact files a user downloads. Recorded in
finding_F773_macos_detached_navigation.md (now 13/13 reproductions).

The macOS cells therefore now run a declared partial stage: install + launcher
resolve + initialize + tools/list (94) + list_instances, and no
navigation
. run_release_gate_journey gained a stages parameter selecting how
far the ONE journey runs - never which journey. This is not an xfail and
not continue-on-error: nothing failing is marked expected-to-fail, and no
cell is silently skipped. A strictly smaller claim is made, and it is named in the
check name (NO-NAVIGATION partial), in a per-cell warning annotation, in the
tool's stdout, in the result record (stages, navigation_verified: false), and
in the known-gaps job (renamed from transport-known-gaps - one home for gap
declarations, now that the same root cause bounds two lanes).

So the gate claims, on macOS/ARM64: the published artifacts install and serve.
It does not claim they navigate.

Topology pins

tests/test_release_workflows.py (14 tests) turns the properties nothing executes
on a normal PR into assertions: publish never builds, publish calls the reusable
gate and re-verifies what it downloads, the aggregate's needs equals every other
job in the file, every edge's result is actually checked, if: always() is intact,
nothing sets continue-on-error, and the macOS cells stay handshake while the
others stay full. Each was proven to bite by temporarily mutating the workflow
and watching the specific test go red.

Observed flake (routed, NOT fixed)

Chrome cold start intermittently fails outright on hosted Ubuntu runners with
nodriver's Failed to connect to browser, after the harness's two warmup attempts
are exhausted. Seen 3 times across 3 runs (transport (Linux/X64) once;
both install-smoke (*/Linux/X64) cells once), each time passing on a plain
re-run with identical code and identical Chrome (150.0.7871.128). It is not
specific to W3 - it hit W1's transport lane too - and it is environmental, not a
packaging defect. Flagged to the team lead for an F-id; deliberately not fixed
here, since the remedy lives in the harness's warmup/retry policy which W3 does
not own.

Incidental find

.claude/worktrees/ (a full repo checkout per agent worktree) was untracked and
un-ignored, so hatchling swept it into the sdist: a local uv build produced a 56 MB
sdist of which 55 MB was that directory. Now gitignored → 14 MB. It also stops ruff
linting three extra copies of the source tree (501 files scanned → 129).

Local verification

ruff format/check, ty (76-diagnostic baseline), vulture, file budgets, and
suppression owners all clean. Unit lane 784 passed (761 pre-existing + 23 new).
actionlint 1.7.7 clean on all three workflows. Both artifact kinds were additionally
smoked end-to-end on Windows against the real uv build output, and all three bite
proofs rehearsed against it, before the first CI round.

AminDhouib and others added 4 commits July 25, 2026 10:31
…l smoke

plan_RELEASE §2.3 (gap G-C), part 1 of 2: the tools the gate jobs call. No
src/ production edits.

tools/package_verify.py — every packaging fact lives here, so build-dist,
package-verify, install-smoke and publish.yml all ask the SAME code rather
than four shell blocks that can drift:
  manifest       hash + describe dist/ once, right after the single build
  verify         re-check a dist/ against that manifest (hashes, sizes,
                 name/version agreement across manifest + both filenames +
                 wheel METADATA + sdist PKG-INFO, wheel/sdist membership,
                 both console entry points, package data)
  hash-check     the cheap one-file precondition a smoke cell / publish runs
  assert-version manifest version == the release tag
Stdlib only, so the jobs that use it need no dependency sync.

embedded/js/*.js are real package data: the cloner engine reads them at
runtime, so a wheel that omits them installs and imports fine and only fails
on first use. verify proves the wheel and the sdist carry byte-identical
copies — comparing the two artifacts to EACH OTHER, not to the git checkout,
so no runner's line-ending policy can make the check lie.

tools/install_smoke.py — installs one artifact into a fresh venv by absolute
path with caches disabled, proves what landed IS the artifact (version, the
package imports from that venv's site-packages, every embedded/js file
matches the hash recorded from the wheel), resolves that env's launcher via
W1's resolve_launcher, and runs W1's canonical journey UNCHANGED. There is
one journey and this is not a second one.

tools/corrupt_artifact.py — the bite-proof fixture. Copy-only by
construction: it refuses to write to its source and re-checks the source
size afterwards, so a negative test can never damage the run's real hashed
artifact.

tests/test_package_verify.py — 23 hermetic pins over synthetic wheels/sdists
(no network, no uv build). Every rule is proven to BITE, not merely to pass:
flipped byte, dropped js member (rejected at manifest time AND by verify),
wheel/sdist package-data disagreement, missing module, missing entry point,
metadata/filename version disagreement, stray .js outside the package,
sdist missing package data, manifest schema version, and the tag
precondition.

.gitignore — found while inspecting the real sdist: .claude/worktrees/ holds
a FULL checkout per agent worktree, and hatchling swept all of it into the
distribution. A local uv build produced a 56 MB sdist of which 55 MB was
that directory; ignoring it gives 14 MB. It also stops ruff linting three
extra copies of the source tree (501 files scanned -> 129).

Local: ruff format/check, ty (76 baseline), vulture, budgets, suppression
owners all clean; unit lane 784 passed (761 pre-existing + 23 new).
Both artifact kinds additionally smoked end-to-end on Windows against the
real uv build output, and the three bite proofs rehearsed against it.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
… files

plan_RELEASE §2.3 (gap G-C), part 2 of 2: the topology. Job semantics stay in
the ONE reusable release-gate.yml; publish.yml CALLS it rather than
re-implementing a second source of truth.

release-gate.yml gains three edges, all wired directly into the stable
`release-gate` aggregate (which already fails on any non-success dependency,
including skipped/cancelled/missing):

  build-dist      the ONLY `uv build` in the run. Validates metadata and
                  package data, hashes everything into release-manifest.json,
                  and uploads dist/ as the run's one immutable artifact.
                  Exposes the built version as a workflow output.
  package-verify  downloads that artifact and re-checks it independently,
                  then runs three IN-JOB BITE PROOFS against throwaway
                  copies (the real artifact is never touched, and no branch,
                  commit, push, tag or publication is involved):
                    1/3 a flipped byte is rejected by the hash check
                    2/3 a wheel missing embedded/js is rejected — the copy is
                        re-hashed into its OWN manifest first, so only the
                        membership rule can reject it, independent of hashing
                    3/3 the publish tag precondition rejects a wrong tag
                  and finally re-verifies that the real artifact is intact.
  install-smoke   {wheel, sdist} x {Ubuntu/X64, Windows/X64, macOS/ARM64}:
                  each cell downloads the artifact, hash-checks it, installs
                  it into a fresh venv by absolute path with caches disabled,
                  and runs W1's canonical journey through the launcher that
                  install produced. No cell rebuilds or fetches from PyPI.

A new `release_tag` input makes the gate itself fail when a tag disagrees
with the built artifact's metadata version, so the disagreement is caught
before any publish job exists to be blocked.

publish.yml is rewritten around the same gate: at the tag SHA it calls
release-gate.yml with `ref` and `release_tag`, so a tag is qualified by
exactly the jobs a pull request runs — including the six install-smoke
cells. `publish` then needs that gate, downloads the already-hashed dist/
from the same run, re-verifies hashes/membership/tag agreement, and uploads
those very files to PyPI and the GitHub release. It never runs `uv build`,
never rebuilds from the checkout, and never re-downloads the version from
PyPI. Any failed, skipped or cancelled cell leaves needs.gate non-success
and nothing is published. The old three-job "unit tests on ubuntu then build
and publish" path is gone; permissions are also narrowed to the publish job.

actionlint 1.7.7 clean on all three workflows (release-gate.yml,
publish.yml, test.yml). The `runner` context is used only at step level,
never in job-level env.

The macOS install-smoke cells are included here deliberately and NOT
pre-excluded: whether F-773 reaches this lane is an empirical question and
this run is the experiment.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
CI round 1 (run 30161880495) answered the empirical question the previous
commit posed. Result: 4 of 6 install-smoke cells green
(wheel+sdist x Linux/X64, Windows/X64); both macOS/ARM64 cells failed with
F-773's exact signature -- about:blank ok in 1.1s/3.1s, a connection to the
CLOSED port 127.0.0.1:1 hung 35.3s, the fixture server served NOTHING, the
network.mojom.NetworkService process alive throughout, workspace already
under RUNNER_TEMP. It is F-773, not a new defect.

New information worth having: this reproduced from a FRESH INSTALL of the
built wheel and sdist into a brand-new venv, not from the editable checkout.
That eliminates install layout / editable install as a factor -- the hang
happens through the exact files a user downloads. Recorded in the finding
(now 13/13 reproductions) along with what the gate does and does not claim.

Rather than drop the cells, they now run a DECLARED PARTIAL stage:

  release_gate_harness.run_release_gate_journey gains `stages`, selecting how
  far the ONE journey runs -- never which journey. "full" (default) is
  unchanged and is what W1's transport test and all four non-macOS smoke
  cells run. "handshake" stops after the non-navigating prefix: initialize ->
  tools/list (94) -> list_instances -> the representative parity call.

So macOS/ARM64 now proves the published artifacts INSTALL AND SERVE, and
proves nothing about navigation. That is a strictly smaller claim, and it is
labelled everywhere it could be misread:
  - the cell name carries "NO-NAVIGATION partial" in the check list
  - a per-cell warning annotation fires before the step runs
  - install_smoke prints "PARTIAL ... must not be reported as full-journey"
  - the result record carries stages + navigation_verified: false
  - the gap-declaration job enumerates full vs partial coverage explicitly

This is deliberately NOT an xfail and NOT continue-on-error: nothing failing
is being marked expected-to-fail, and no cell is silently skipped while the
gate reports green. A smaller thing is run, and it is named.

transport-known-gaps -> known-gaps: the same root cause now bounds a second
lane, and gap declarations get ONE home rather than two. Aggregate edge
renamed with it.

Local: transport marker still passes (the full journey is unchanged), unit
lane 784 passed, ruff/ty/vulture/budgets/suppression-owners clean,
actionlint clean. Handshake mode exercised end-to-end on Windows.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
…ress

The W3 properties most likely to rot are the ones nothing executes on a
normal PR. A tag build happens rarely and cannot be dry-run from here, so
"publish never rebuilds" and "the aggregate never forgets an edge" were only
review promises. tests/test_release_workflows.py turns them into 14
structural pins over the workflow YAML.

Build once / publish the same files:
  - `uv build` (or any build command) appears in build-dist and NOWHERE else
    in the gate
  - publish.yml never builds and never re-installs the package from PyPI
  - publish.yml CALLS the reusable gate with both `ref` and `release_tag`
    instead of duplicating job semantics
  - the publish job needs that gate, downloads exactly the one `dist`
    artifact, and re-verifies it before uploading

Aggregate integrity:
  - release-gate's `needs` equals every other job in the file — a lane that
    exists but is not an edge can fail while the required check stays green.
    This is what will tell W5 to wire `release-evidence` rather than letting
    it discover the omission later.
  - every listed edge's `.result` is actually asserted in the script
  - the aggregate keeps `if: always()` (a failed dependency must FAIL it, not
    skip it — a skipped required check does not block merge)
  - no job or step anywhere in the gate sets continue-on-error

Declared gap stays declared:
  - install-smoke covers wheel AND sdist on all three cells
  - macOS/ARM64 is `handshake`, Linux/Windows are `full`
  - the partial cells carry "NO-NAVIGATION partial" in the check name
  - known-gaps exists, is a required edge, and names both bounded lanes
  - the PR caller has no path filter that could omit packaging changes

Each pin was proven to BITE by temporarily mutating the workflow and
watching the specific test go red (dropped aggregate edge -> failure;
macOS flipped to full -> failure; `uv build` added to publish -> failure),
then restoring from git.

pyyaml moves from a transitive dependency to a declared TEST extra so the
pin cannot vanish if uvicorn[standard] drops it. Test-only; the lockfile
change is 2 lines and no version churn.

Local: 798 passed (784 + 14), ruff/ty/vulture/budgets/suppression-owners
and actionlint all clean.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant