[STAGING - DO NOT MERGE] release integration tree for W5 evidence generation - #51
Open
AminDhouib wants to merge 12 commits into
Open
AminDhouib wants to merge 12 commits into
AminDhouib wants to merge 12 commits into
Conversation
…al detectors (G-D) plan_RELEASE W4. Lands the deterministic offline stealth-invariant gate plus the non-gating informational online tier, asserting the product's headline "undetectable" promise (G-D) that was previously asserted nowhere. - tests/fixture_app/stealth_probe.html: armed passive signal collector served by the ONE fixture mechanism (release_gate_harness.serve_fixture_app). Publishes a single closed-schema window.__STEALTH_PROBE_RESULT__ and one completion event; collects nothing until explicitly released. - tests/test_stealth.py: versioned predicate table; closed-schema validator with deterministic negative controls; per-collector-family sensitivity controls; the exact ordered CDP prerequisite transcript (Runtime.enable -> Page.enable -> Network.enable -> DOM.getDocument -> Runtime.evaluate(nonce) -> Page.captureScreenshot) driven via the project's own tab.send(uc.cdp.*) seam; process-flag evidence with W2's exact-binary identity; a redacted result artifact; and a real vanilla-control that must be detected. The vanilla control is the SAME product spawn path with the stealth arg-filter neutralized (the one intentional treatment) -- no second Chrome-launch path. - pyproject.toml: adds the `stealth` (gating) and `online` (non-gating) markers. Gate lane selects `-m "stealth and not online"`; unit lane `-m "not integration"` excludes the whole module. Zero src/ edits. One product predicate does not pass and is pinned honestly as a strict xfail (F-770): under headless the product UA still advertises "HeadlessChrome"; nodriver's default stealth does not mask it. An xfailed invariant does NOT satisfy the stealth release claim -- the headless UA vector remains detectable and is flagged for a fix owner. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
FIX-C shipped on a strongly evidenced hypothesis that was wrong, so FIX-D
measures before it fixes. "The UA leaks" is not precise enough to act on:
a pre-launch --user-agent= flag and a post-launch Emulation.setUserAgentOverride
cover different subsets of the surface, so D0 records each vector separately on
every qualified cell BEFORE D1 picks a mechanism.
V1 navigator.userAgent page
V2 the HTTP User-Agent header the fixture wire -- what the server READ, not
server actually received what the page claims
V3 userAgentData brands + high entropy page
V4 Browser.getVersion() -> userAgent CDP
V2 is new evidence: a page cannot read its own request headers, so the armed
probe issues one same-origin POST to the fixture app's existing /api/echo route,
which reflects the headers it read off the wire. A page-only assertion can pass
while the real header still leaks.
test_f770_ua_vectors_are_measured asserts only that every vector is OBSERVABLE
(a vector that silently stops being collected would make the F-770 signals
vacuous without turning anything red); the leak verdict stays in GATE_SIGNALS
and the strict xfail. Both product and control readings land in the release
artifact, which is also the differential D2 will need.
Also renames the artifact-dir knob off the STEALTH_MCP_* namespace: settings.
_reject_unknown_prefixed_env fails get_settings() for any unknown key there, so
exporting STEALTH_MCP_STEALTH_ARTIFACT_DIR would have detonated the backend the
moment CI used it.
fix-d-stealth-cells.yml is temporary scaffolding: the three-cell offline-stealth
edge lives in W2's release-gate.yml, which is not in this branch's ancestry, and
FIX-D needs the measurement on all three cells. Same selector as W2's edge;
delete when release-gate.yml reaches this history.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
…l smoke
plan_RELEASE §2.3 (gap G-C), part 1 of 2: the tools the gate jobs call. No
src/ production edits.
tools/package_verify.py — every packaging fact lives here, so build-dist,
package-verify, install-smoke and publish.yml all ask the SAME code rather
than four shell blocks that can drift:
manifest hash + describe dist/ once, right after the single build
verify re-check a dist/ against that manifest (hashes, sizes,
name/version agreement across manifest + both filenames +
wheel METADATA + sdist PKG-INFO, wheel/sdist membership,
both console entry points, package data)
hash-check the cheap one-file precondition a smoke cell / publish runs
assert-version manifest version == the release tag
Stdlib only, so the jobs that use it need no dependency sync.
embedded/js/*.js are real package data: the cloner engine reads them at
runtime, so a wheel that omits them installs and imports fine and only fails
on first use. verify proves the wheel and the sdist carry byte-identical
copies — comparing the two artifacts to EACH OTHER, not to the git checkout,
so no runner's line-ending policy can make the check lie.
tools/install_smoke.py — installs one artifact into a fresh venv by absolute
path with caches disabled, proves what landed IS the artifact (version, the
package imports from that venv's site-packages, every embedded/js file
matches the hash recorded from the wheel), resolves that env's launcher via
W1's resolve_launcher, and runs W1's canonical journey UNCHANGED. There is
one journey and this is not a second one.
tools/corrupt_artifact.py — the bite-proof fixture. Copy-only by
construction: it refuses to write to its source and re-checks the source
size afterwards, so a negative test can never damage the run's real hashed
artifact.
tests/test_package_verify.py — 23 hermetic pins over synthetic wheels/sdists
(no network, no uv build). Every rule is proven to BITE, not merely to pass:
flipped byte, dropped js member (rejected at manifest time AND by verify),
wheel/sdist package-data disagreement, missing module, missing entry point,
metadata/filename version disagreement, stray .js outside the package,
sdist missing package data, manifest schema version, and the tag
precondition.
.gitignore — found while inspecting the real sdist: .claude/worktrees/ holds
a FULL checkout per agent worktree, and hatchling swept all of it into the
distribution. A local uv build produced a 56 MB sdist of which 55 MB was
that directory; ignoring it gives 14 MB. It also stops ruff linting three
extra copies of the source tree (501 files scanned -> 129).
Local: ruff format/check, ty (76 baseline), vulture, budgets, suppression
owners all clean; unit lane 784 passed (761 pre-existing + 23 new).
Both artifact kinds additionally smoked end-to-end on Windows against the
real uv build output, and the three bite proofs rehearsed against it.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
… files
plan_RELEASE §2.3 (gap G-C), part 2 of 2: the topology. Job semantics stay in
the ONE reusable release-gate.yml; publish.yml CALLS it rather than
re-implementing a second source of truth.
release-gate.yml gains three edges, all wired directly into the stable
`release-gate` aggregate (which already fails on any non-success dependency,
including skipped/cancelled/missing):
build-dist the ONLY `uv build` in the run. Validates metadata and
package data, hashes everything into release-manifest.json,
and uploads dist/ as the run's one immutable artifact.
Exposes the built version as a workflow output.
package-verify downloads that artifact and re-checks it independently,
then runs three IN-JOB BITE PROOFS against throwaway
copies (the real artifact is never touched, and no branch,
commit, push, tag or publication is involved):
1/3 a flipped byte is rejected by the hash check
2/3 a wheel missing embedded/js is rejected — the copy is
re-hashed into its OWN manifest first, so only the
membership rule can reject it, independent of hashing
3/3 the publish tag precondition rejects a wrong tag
and finally re-verifies that the real artifact is intact.
install-smoke {wheel, sdist} x {Ubuntu/X64, Windows/X64, macOS/ARM64}:
each cell downloads the artifact, hash-checks it, installs
it into a fresh venv by absolute path with caches disabled,
and runs W1's canonical journey through the launcher that
install produced. No cell rebuilds or fetches from PyPI.
A new `release_tag` input makes the gate itself fail when a tag disagrees
with the built artifact's metadata version, so the disagreement is caught
before any publish job exists to be blocked.
publish.yml is rewritten around the same gate: at the tag SHA it calls
release-gate.yml with `ref` and `release_tag`, so a tag is qualified by
exactly the jobs a pull request runs — including the six install-smoke
cells. `publish` then needs that gate, downloads the already-hashed dist/
from the same run, re-verifies hashes/membership/tag agreement, and uploads
those very files to PyPI and the GitHub release. It never runs `uv build`,
never rebuilds from the checkout, and never re-downloads the version from
PyPI. Any failed, skipped or cancelled cell leaves needs.gate non-success
and nothing is published. The old three-job "unit tests on ubuntu then build
and publish" path is gone; permissions are also narrowed to the publish job.
actionlint 1.7.7 clean on all three workflows (release-gate.yml,
publish.yml, test.yml). The `runner` context is used only at step level,
never in job-level env.
The macOS install-smoke cells are included here deliberately and NOT
pre-excluded: whether F-773 reaches this lane is an empirical question and
this run is the experiment.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
D0's first three-cell run measured Windows/X64 and failed to launch Chrome at
all on Linux/X64 and macOS/ARM64: "Failed to connect to browser", ~3s after the
first spawn. Cause is the lane, not the product. The gate selector
`-m "stealth and not online"` selects test_stealth.py ALONE, so — unlike the
full `-m integration` lane where earlier E2E modules already paid for it —
nobody has absorbed Chrome's cold start, and nodriver gives the debug port only
a few seconds. This is why the same tests are green in W4's ubuntu integration
lane and red here.
Fix is the shared idempotent warmup e2e_helpers already owns for exactly this
("the first Chrome launch on CI is slow / flaky"), not a widened timeout. Linux
also gets Xvfb + DISPLAY, mirroring W2's integration cell.
NOTE for W2/W5: W2's release-gate `offline-stealth` edge runs this same isolated
selector, so it inherits the same cold-start exposure the moment W4's tests land
under it. This commit fixes it at the module, which that edge picks up for free.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
CI round 1 (run 30161880495) answered the empirical question the previous commit posed. Result: 4 of 6 install-smoke cells green (wheel+sdist x Linux/X64, Windows/X64); both macOS/ARM64 cells failed with F-773's exact signature -- about:blank ok in 1.1s/3.1s, a connection to the CLOSED port 127.0.0.1:1 hung 35.3s, the fixture server served NOTHING, the network.mojom.NetworkService process alive throughout, workspace already under RUNNER_TEMP. It is F-773, not a new defect. New information worth having: this reproduced from a FRESH INSTALL of the built wheel and sdist into a brand-new venv, not from the editable checkout. That eliminates install layout / editable install as a factor -- the hang happens through the exact files a user downloads. Recorded in the finding (now 13/13 reproductions) along with what the gate does and does not claim. Rather than drop the cells, they now run a DECLARED PARTIAL stage: release_gate_harness.run_release_gate_journey gains `stages`, selecting how far the ONE journey runs -- never which journey. "full" (default) is unchanged and is what W1's transport test and all four non-macOS smoke cells run. "handshake" stops after the non-navigating prefix: initialize -> tools/list (94) -> list_instances -> the representative parity call. So macOS/ARM64 now proves the published artifacts INSTALL AND SERVE, and proves nothing about navigation. That is a strictly smaller claim, and it is labelled everywhere it could be misread: - the cell name carries "NO-NAVIGATION partial" in the check list - a per-cell warning annotation fires before the step runs - install_smoke prints "PARTIAL ... must not be reported as full-journey" - the result record carries stages + navigation_verified: false - the gap-declaration job enumerates full vs partial coverage explicitly This is deliberately NOT an xfail and NOT continue-on-error: nothing failing is being marked expected-to-fail, and no cell is silently skipped while the gate reports green. A smaller thing is run, and it is named. transport-known-gaps -> known-gaps: the same root cause now bounds a second lane, and gap declarations get ONE home rather than two. Aggregate edge renamed with it. Local: transport marker still passes (the full journey is unchanged), unit lane 784 passed, ruff/ty/vulture/budgets/suppression-owners clean, actionlint clean. Handshake mode exercised end-to-end on Windows. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
… F-774 D0 measured, on all three qualified cells under headless=True, Chrome 150: V1 navigator.userAgent .................. LEAK Linux, Windows, macOS V2 HTTP User-Agent header the server READ LEAK Linux, Windows, macOS V3 userAgentData brands / sec-ch-ua ..... clean Linux, Windows, macOS V4 Browser.getVersion().userAgent ....... LEAK Linux, Windows, macOS Two of the plan's stated assumptions were wrong, which is what D0 exists for: V3 does NOT leak (brands are identical headless and headed everywhere), and V4 IS covered by --user-agent= (the flag is process-wide and Browser.getVersion follows it). So M-A alone covers every leaking vector; M-B is not implemented. --user-agent-product= was measured too and is inert in release Chrome. D1. merge_browser_args now supplies a masked default --user-agent= when the caller chose none. Placement is deliberate: * It is a LAUNCH FLAG, so it is process-wide: every tab, worker and subresource, including tabs the page itself opens, and it reaches the real HTTP request header. A per-target Emulation override would satisfy the spawn tab and leak on the next one -- the failure mode FIX-C's create_hook re-arm existed to fix. * It lives in merge_browser_args because that is the seam W4's vanilla control neutralizes (D2/§4). Masking anywhere else would have masked the CONTROL too, vanilla_detected would have gone false, and the whole stealth suite would have silently become vacuous. The signature is unchanged, so the existing control disables the new behaviour with no edit -- and the differential is asserted, not assumed. * The mask is CONSTRUCTED, not guessed: Chrome's reduced UA froze the platform token and the WebKit/Safari tokens, leaving only <major>.0.0.0 to resolve, so the string is byte-identical to what the same binary emits headed. The three platform tokens are the ones measured on the runners themselves. Version resolution is lru_cached per executable, and Windows reads the version-named sibling directory rather than shelling out (chrome.exe --version hands the flag to a running Chrome instead of printing). Unresolvable version or unknown platform => no mask: a UA that contradicts sec-ch-ua would be a worse tell than the honest headless one. Zero net lines in browser_manager.py, which sits exactly at its LOC cap. D2/D3. ua_no_headless_token moves from XFAIL_SIGNALS into the gating table with three more rows -- the wire header, page-vs-wire equality, and UA-major-vs- client-hints coherence -- and the strict xfail is DELETED, not relaxed. Pinned against a real browser: a tab created later through new_tab, an explicit caller user_agent still winning, V4 at the CDP level, and the differential that the control's UA still contains HeadlessChrome while the product's is that same string minus the token. F-774, opened by this fix and measured not assumed: a --user-agent override makes Chrome blank the high-entropy UA client hints (architecture, bitness, platformVersion, uaFullVersion, fullVersionList). brands/mobile/platform -- and therefore every sec-ch-ua* header actually on the wire -- stay correct, so the residue is JS-only and strictly smaller than the server-side substring test it replaced. Recorded as a strict xfail with the before/after and routed in audit/stage2/finding_F774_ua_client_hints_high_entropy_blanked.md; W5's contract must qualify the claim accordingly. Fixing it needs userAgentMetadata, which is per-target -- a different mechanism, deliberately out of scope. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
…ress
The W3 properties most likely to rot are the ones nothing executes on a
normal PR. A tag build happens rarely and cannot be dry-run from here, so
"publish never rebuilds" and "the aggregate never forgets an edge" were only
review promises. tests/test_release_workflows.py turns them into 14
structural pins over the workflow YAML.
Build once / publish the same files:
- `uv build` (or any build command) appears in build-dist and NOWHERE else
in the gate
- publish.yml never builds and never re-installs the package from PyPI
- publish.yml CALLS the reusable gate with both `ref` and `release_tag`
instead of duplicating job semantics
- the publish job needs that gate, downloads exactly the one `dist`
artifact, and re-verifies it before uploading
Aggregate integrity:
- release-gate's `needs` equals every other job in the file — a lane that
exists but is not an edge can fail while the required check stays green.
This is what will tell W5 to wire `release-evidence` rather than letting
it discover the omission later.
- every listed edge's `.result` is actually asserted in the script
- the aggregate keeps `if: always()` (a failed dependency must FAIL it, not
skip it — a skipped required check does not block merge)
- no job or step anywhere in the gate sets continue-on-error
Declared gap stays declared:
- install-smoke covers wheel AND sdist on all three cells
- macOS/ARM64 is `handshake`, Linux/Windows are `full`
- the partial cells carry "NO-NAVIGATION partial" in the check name
- known-gaps exists, is a required edge, and names both bounded lanes
- the PR caller has no path filter that could omit packaging changes
Each pin was proven to BITE by temporarily mutating the workflow and
watching the specific test go red (dropped aggregate edge -> failure;
macOS flipped to full -> failure; `uv build` added to publish -> failure),
then restoring from git.
pyyaml moves from a transitive dependency to a declared TEST extra so the
pin cannot vanish if uvicorn[standard] drops it. Test-only; the lockfile
change is 2 lines and no version churn.
Local: 798 passed (784 + 14), ruff/ty/vulture/budgets/suppression-owners
and actionlint all clean.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Staging branch only - NOT part of the PR merge queue. Gives plan_RELEASE W5 a single foundation containing W1-W4 plus FIX-B/C/E/F and W3, so the generated release contract describes one commit instead of nine unmerged branches. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Staging branch only - NOT part of the PR merge queue. Completes the single foundation for plan_RELEASE W5: W1-W4 plus FIX-B/C/D/E/F and W3 in one commit, so the generated release contract describes one tree rather than nine unmerged branches. Conflict resolved in pyproject.toml: W3 added `pyyaml` to the `test` extra and FIX-D added the `online` pytest marker. Independent additions in different sections; both kept. Dropped .github/workflows/fix-d-stealth-cells.yml. FIX-D added it as explicitly temporary scaffolding because release-gate.yml was not in its ancestry; it runs W2's offline-stealth selector verbatim. That condition is now satisfied - the real gate is here - so the scaffolding would be a second way to run one lane. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Merging the W4/FIX-D line puts the stealth tests into this history for the first time, which changes what the offline-stealth job means. Two defects follow, both of which would have produced a GREEN lane that proves nothing: 1. Exit 5 (zero collected) was treated as success. That was correct while W4's tests were absent, but the selector now collects 23 tests. Left as-is, a marker rename that silently stopped collecting them would keep the lane green. Removed: any non-zero now fails the job and the gate. 2. The job had no Chrome support at all - no Xvfb, no DISPLAY, timeout 180 - because an empty lane never launched a browser. FIX-D's temporary fix-d-stealth-cells.yml scaffolding DID have them; dropping that scaffolding without porting them would have broken the Linux cell. Ported Xvfb, the Linux-only DISPLAY, the Chrome-identity step, and the 300s timeout, mirroring the integration cell. DISPLAY is set at STEP level: the `runner` context is not available in job-level `env:` and would silently evaluate to empty. actionlint clean. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
The known install-smoke flake stopped being occasional: one CI round took
`transport (Linux/X64)` AND `install-smoke (wheel Linux/X64)` down on BOTH open
PRs at once, blocking the release. Diagnosed rather than re-run:
ToolError: Failed to spawn browser: Failed to connect to browser
--- warmup / nav probe ---
{"attempted": true, "ok": false, "attempts": 2, ...}
Both warmup attempts were exhausted, so the journey then met a cold Chrome and
failed its own spawn.
"Failed to connect to browser" is nodriver giving up on the CDP endpoint - a
FAST connect failure, not a timeout. Raising WARMUP_TIMEOUT cannot help. The
retry loop had no backoff at all, so attempt 2 fired straight back into the
resource contention that had just failed attempt 1.
WARMUP_ATTEMPTS 2 -> 4, with WARMUP_BACKOFF_SECONDS * attempt between tries -
the same shape e2e_helpers.warmup_once already uses for the in-process warmup.
Deliberately NOT touching the journey's own spawn. The warmup is best-effort and
asserts nothing, so retrying it masks nothing; adding a retry around the
journey's spawn would let a genuine spawn regression pass the gate. If warmup
succeeds, the journey's spawn is warm - that is the whole point of it.
This lands on the shared base so both open PRs inherit it.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
This exists for one reason: plan_RELEASE W5 generates a release contract from
current-SHA evidence, and that evidence is currently spread across nine unmerged
branches. W5 needs one tree to describe. This branch is that tree; it is not a
competing path to
main.The human merge queue is unchanged and still authoritative:
#42 → #43 → #44 → #46 → #49 → #50, with#45 → #47on the W4 line and#48behind
#46.What it is
True merge commits (never squash) of the three live lines:
One conflict, in
pyproject.toml: W3 addedpyyamlto thetestextra, FIX-Dadded the
onlinepytest marker. Independent additions in different sections;both kept.
Two false-greens the merge surfaced
Both were correct on their own branch and only became wrong once the lines met.
Neither was reachable from any single branch, which is the argument for doing this
integration at all.
1.
offline-stealthtreated pytest exit 5 (zero collected) as success. Rightwhile W4's stealth tests were absent; the selector now collects 23 tests. Left
alone, any future marker rename would have produced a permanently green lane
asserting nothing. Removed — any non-zero now fails the job and the gate.
2. That lane had no Chrome support at all — no Xvfb, no
DISPLAY, timeout 180— because an empty lane never launches a browser. FIX-D's deliberately temporary
fix-d-stealth-cells.ymlscaffolding did have all of it. Dropping thescaffolding (its stated exit condition —
release-gate.ymlis now in thishistory) without porting those would have broken the Linux stealth cell. Ported
Xvfb, the Linux-only
DISPLAY, the Chrome-identity step, and the 300s timeout,mirroring the
integrationcell.DISPLAYis set at step level, not job level: therunnercontext is notavailable in job-level
env:and evaluates to empty there.actionlintclean onall three workflows.
Verification
pytest -m "not integration") green on the merged tree.suite, FIX-D's UA mask, FIX-E's
list_tabsfix and FIX-F's_find_tabare allpresent; the scaffolding is gone; all six FIX plans are in place.
no individual branch has ever exercised.
What this branch does NOT claim
Unchanged from the individual branches, and W5 must reproduce these bounds rather
than soften them:
xfailed (F-773). An xfail would let green imply macOS coverage.
NO-NAVIGATION partial— handshakeprefix only,
navigation_verified: falsein the journey record. The claim is"this artifact installs and serves", not "the journey passes on macOS".
the wire header, and CDP; F-774 (blanked high-entropy client hints) is open and
must be qualified in the contract.
get_cookiesis still unresolved, so no 94-tool claim is licensed yet.