RELEASE-2 W2: required Ubuntu-x64 / Windows-x64 / macOS-arm64 release gate (G-B) - #44
Open
AminDhouib wants to merge 11 commits into
Open
AminDhouib wants to merge 11 commits into
AminDhouib wants to merge 11 commits into
Conversation
…vidence Add the two single-source resolvers the three-OS gate runs before any test work, plus the platform assertions the gate newly exercises on Windows/macOS: - tools/runner_identity.py: fail the job unless (runner.os, runner.arch) is a qualified release cell AND matches the matrix cell's declared pair; record image OS/version + python as a JSON identity artifact. A label/arch migration is a RED gate, not an invisible change. - tools/resolve_chrome.py: the ONE source of the expected image Chrome Stable identity (path + version + CDP product), Chrome-Stable-only candidates per OS so Chromium/Edge/another channel surfaces as a mismatch. - tests/test_platform_lifecycle.py: exercise the REAL msvcrt/fcntl lock branch and its contention semantics (asserted nowhere — every prior singleton test monkeypatches the lock away). - test_browser_integration.py: repeated spawn/close leak guard, and the three-way image-Chrome-Stable identity check (auto-discovery == resolver == launched-binary Browser.getVersion). - test_clone_storage / test_singleton_backend_logging: assert the os.name session-root default and the OS-branch detach flags on the platform that ships each branch. - pyproject.toml: register the `stealth` marker for W2's offline-stealth edge (W4 lands the tests). Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
…G-B) Land the ONE reusable release gate (.github/workflows/release-gate.yml, workflow_call) and make test.yml a thin caller — job semantics live in the reusable workflow only, never duplicated into a second YAML source of truth (W3's publish.yml will call the same workflow for tags). - Qualified cells: Ubuntu/X64, Windows/X64, macOS/ARM64, asserted per job via tools/runner_identity.py with an uploaded identity artifact; a label/arch migration is a red gate. - Per-OS unit (py3.11-3.13) + coverage; coverage uses the PATH form --cov=src/stealth_chrome_devtools_mcp (the module form falsely reports ~53%), same floor 55, each OS uploads its own report. - Per-OS integration + transport assert the image Chrome Stable identity via tools/resolve_chrome.py + CDP Browser.getVersion; no continue-on-error; Linux alone gets Xvfb for headed cases and headless never inherits DISPLAY=:99. - offline-stealth wires the (empty until W4) `stealth and not online` lane; pytest exit 5 (0 collected) is treated as pass, any real failure fails red. - Stable aggregate `release-gate` (if: always()) directly needs quality, unit-tests, coverage, integration, transport, offline-stealth and is red unless every edge is success. W3 adds build-dist/package-verify/install-smoke and W5 adds release-evidence to this same needs list — the aggregate is never replaced. - All actions pinned by full commit SHA. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
…alid at job level) actionlint [expression]: the runner context is not available in job-level env (only github/inputs/matrix/needs/secrets/strategy/vars) — runner.temp would never resolve there. Step-level env on the two pytest steps that need the isolated session root; behavior identical, now actually effective. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
…de only) 1) resolve_launcher followed the POSIX venv python symlink out of the venv (Path.resolve) into the uv base-interpreter bin, where no entry points live — transport failed on Linux/macOS while Windows (real exe, no symlink) passed. Now Path.absolute(): absolute, symlinks untouched. 2) test_tabs_lifecycle: the known close-path flake has a second signature — during target-detach propagation nodriver's table can hold a half-detached entry so list_tabs RAISES TypeError (object Connection can't be used in 'await'), seen on the Linux and Windows CI images. Extended the existing bounded-poll to tolerate the transient raise within the 10s settle window, per plan_RELEASE SS2.8 (bounded-wait in the test harness, never src; xfail-quarantine banned). Final list_tabs must still return cleanly with the tab gone — teeth preserved. Local (Windows): both tests pass; ruff format+check clean. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
…test-side only)
Round 2 left macOS as the only red cell, with two faces of one cause at the
same SHA: transport spawn_browser died at ~9s (nodriver 'Failed to connect
to browser') while the integration job's spawn succeeded and its FIRST
navigate hit the product's 35s CDP deadline. Meanwhile 54 in-process Chrome
tests passed on that same image -- every one of them behind
e2e_helpers.warmup_once ('the first Chrome launch on CI is slow / flaky').
The journey was the only Chrome test launching cold: a fresh
browser-session root pays Chrome cold start PLUS the master-profile
bootstrap clone-on-spawn needs.
- _cold_start_warmup: the wire-level twin of warmup_once (same reason, same
best-effort contract). Not a second convention -- the journey drives a
separate backend process and cannot share the in-process warmup. Outcome
is recorded (cold_start_warmup.ok) and never fatal; the canonical journey
stays the sole evidence, asserted in full.
- _isolated_env: create the macOS ~/Library skeleton, the same coherent-home
rule the Windows AppData skeleton follows (macOS has no env var to point
at Application Support -- the dirs ARE the coherence).
- _headless_spawn_kwargs: one home for headless+sandbox spawn kwargs, now
that two call sites need them.
Product timeouts deliberately untouched: tuning the CDP deadline to pass CI
would mask a user-facing latency, not fix it.
Local (Windows): transport 1 passed; warmup verified live as
{'attempted': true, 'ok': true}; ruff + suppression-owners clean.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
…chain on macOS Round 3 narrowed both remaining reds to one root cause each. macOS transport (deterministic -- both times it got past spawn): navigate burned the product's full 35s CDP deadline. Two contributors, both fixed: - the warmup spawned and closed but never LOADED a page, so the journey's first navigate was still cold (network service, renderer, font cache). Warmup now navigates as well. - a throwaway HOME on macOS has no login keychain, so Chrome's Safe Storage lookup has nothing to talk to and blocks. --use-mock-keychain (what Puppeteer ships by default for this) is scoped to the isolated-home harness -- the macOS sibling of the existing AppData/Library skeletons. Linux integration (flake; passed rounds 1-2): the cold launch does not just run slow, it intermittently fails outright -- on that run it defeated the module's own warmup AND then the first real test. All three warmups (wire-level, e2e_helpers, test_browser_integration) now retry once. Warmup asserts nothing, so a retry buys reliability at zero evidence cost; the real tests keep their single-shot assertions. Deliberately NOT done: raising the product's 35s CDP deadline to pass CI -- that masks user-facing latency instead of fixing its cause. Local (Windows): transport + browser-integration 19 passed; warmup verified live as ok=true attempts=1; ruff + suppression-owners clean. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
…s on failure Linux: my round-2 bounded wait expired with its own assertion -- which disproved the premise the original comment was built on. Read from nodriver 0.47 source rather than inferred: Browser.update_targets() APPENDS raw Connection objects for newly discovered targets (browser.py:574-584) and Browser.tabs returns them despite being annotated List[Tab]; only Tab defines __await__ (tab.py:1262). So browser_manager.list_tabs's `await tab` (browser_manager.py:1307) raises TypeError for any DISCOVERED target, and the bad entry PERSISTS -- it is not the transient detach race the test assumed, so no bounded wait could ever clear it. That is a real defect in one of the 94 tools: list_tabs can raise a bare TypeError (also off-convention -- not a ToolError) after any tab close. plan_RELEASE forbids src edits, so per SS2.8 this takes the other allowed branch: characterization pin + route as F-771, flagged as a KNOWN GAP so a green run never reads as "close_tab's post-state is verified". Teeth kept: close_tab's own contract still asserted hard, a successful listing must not contain the closed tab, and the ONLY tolerated deviation is this one pinned TypeError -- any other exception still fails. macOS: navigate still burns the 35s CDP deadline, so the keychain/page-load theory is not the cause. Stop inferring: failures now dump the isolated backend's OWN log files (the evidence that root-caused B1), which otherwise die with the throwaway home. Read only after a failure, so the happy path pays nothing. Local (Windows): tabs-lifecycle + transport 2 passed (the xfail branch is NOT taken where list_tabs works, so the pin cannot mask a healthy platform); log collector unit-checked; ruff clean. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
…trips
The backend logs from the last round paid for themselves twice.
1. --use-mock-keychain never reached Chrome: the product's own stealth filter
removed it ("Stripped 1 detectable arg(s): --use-mock-keychain stripped:
Playwright default"). Round 4 therefore tested nothing, and the flag would
have stayed as an inert arg that trips a stealth warning on every spawn.
Removed, with the reason recorded where the next reader will look.
2. The real macOS shape: EVERY navigate (warmup and journey alike) fails --
tab.get(url) exceeds the product's 30s internal deadline, the tab is
replaced, and the tool ends at its 35s CDP deadline. Not a cold-start
effect: the second attempt, on an already-warm Chrome, fails identically.
The next question is binary and cannot be answered from the tool boundary,
where both causes look the same: did Chrome ever REACH the fixture server
(then the page load stalls in CDP) or not (then it is network/launch)? So the
fixture server now records the request lines it actually served, surfaced in
the failure text. Bounded at 50, diagnostics only, asserted on by nobody.
Local (Windows): transport 1 passed; hit recording verified live
(GET /index.html, GET /api/json); ruff clean.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
… stall Round 6 answered the binary question: the fixture server served NOTHING, so Chrome never issued the request -- while the same backend log shows CDP working fine on that instance (viewport set, hooks installed). The request dies before it leaves the browser. That points at the paused-request path. With ZERO hooks configured, dynamic_hook_system.setup_interception still enables catch-all Fetch at BOTH the REQUEST and RESPONSE stages (dynamic_hook_system.py:274-285), so Chrome pauses EVERY request until _on_request_paused resumes it. Pausing happens before the request goes out -- which is exactly a silent server plus a 30s tab.get() timeout. This probe discriminates it: about:blank makes no NETWORK request, so Fetch has nothing to pause; a real URL does. blank-ok + http-hang would confirm the paused-request path and rule out the tab, the CDP session and reachability, none of which the tool boundary can tell apart. Recorded in the warmup record and printed on failure; best-effort, asserted on by nobody. Local (Windows, both work): about_blank ok 1.1s, http ok 0.6s. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
…he cold-launch retry
macOS, confirmed by the probe: about:blank navigates OK in 1.2s while the
fixture URL times out at 35.4s with the server seeing NOTHING. That clears
the tab, the CDP session, and cold start -- all of which about:blank
exercises -- and isolates the stall to the path only a NETWORK request takes.
Two candidates remain and they lead to opposite conclusions, so this round
separates them. Navigating to a port nothing listens on gets an immediate
ECONNREFUSED and an error page:
- if REFUSED also hangs, the request never reached the network stack, i.e.
it is sitting in the catch-all Fetch pause the product enables on every
spawn even with zero hooks -> a product bug (src fix, not W2's to make).
- if REFUSED errors fast while the fixture port hangs, the network stack is
fine and something drops that connection -> a runner/environment problem
(mine to fix, in the harness or the workflow).
Linux: the cold-launch flake took out the first TWO tests of
test_browser_integration this time, so 2 attempts was not enough. Now 3 with
a 2s/4s backoff -- an immediate retry just hits the same busy machine, so the
wait is the point. Later tests in that file passed both times, which is what
says "cold machine", not "broken code".
Local (Windows): transport + browser-integration 19 passed; ruff clean.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
…OS navigation) Triage + fix design for the defect W2's macOS cells exposed over rounds 1-8. Human authorized the FIX-plan route 2026-07-24 over routing macOS as a gap. Evidence is a probe, not inference: about:blank navigates fine (1.1s) while a connection to a port nothing listens on ALSO hangs 35s -- and a refused connection cannot hang, so the request never reached the network stack. It is paused in the catch-all Fetch interception the product enables on every spawn even with zero hooks, and the resume handler never fires on macOS (no _on_request_paused line in any failing backend log). The plan records one design trap found while writing it: create_hook does NOT re-arm interception, so a bare early-return would silently break spawn -> create_hook -> navigate. C1 therefore ships the skip and the re-arm together or not at all. Scope is stated honestly: this restores the DEFAULT zero-hook path on macOS. Why the handler never dispatches under the detached backend stays open as F-773, and the hooks-on-macOS path remains unproven and unclaimed. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Member
Author
|
The three-OS gate is green — 23/23, including the Coverage as it actually stands:
Getting here took 11 CI rounds and surfaced four genuine defects, none of which the pre-existing in-process suite could see: B1 (per-session lifespan killing every browser over the wire — fixed in #43), F-771 ( What this gate does NOT verify, by design: macOS navigation. The macOS transport cell is excluded rather than xfail-quarantined, and Still human-gated before W2's DoD closes: the repository ruleset requiring only |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
RELEASE-2 (W2) — stacked on #43 (audit/release-fix-b)
The entire gate now lives in ONE reusable workflow,
.github/workflows/release-gate.yml(workflow_call);test.ymlis a thin caller (W3'spublish.ymlwill call the same file for tags with an exact-SHArefinput). The stable required check for a repository ruleset is the aggregate jobrelease-gate.Qualified cells: Linux/X64, Windows/X64, macOS/ARM64 — every job asserts its cell in-job via
tools/runner_identity.py(red on unqualified pair or cell mismatch) and uploads an identity artifact. Real-Chrome jobs additionally record the image's Chrome Stable identity via CDPBrowser.getVersion(tools/resolve_chrome.py).Edges (all three OSes unless noted):
quality(ubuntu, OS-independent) ·unit-tests(×py3.11–3.13) ·coverage(path-form--cov=src/…, floor 55, per-OS reports — the module form falsely reports ~53%) ·integration·transport(the W1 real-stdio journey, per-OS) ·offline-stealth(W2 wires the lane-m "stealth and not online"; empty today, exit-5 treated as pass, any real failure still gates; W4 lands the tests) ·release-gateaggregate (if: always(), red unless every edge is exactlysuccess).New per-OS evidence tests (all additive, zero
src/edits): singleton lock contention (realmsvcrt/fcntl), detached-spawn flags, per-OS session-root default, backend-logging paths.Local gate (Windows): unit 758 passed (753 baseline + 5 new) · ruff/ty(76)/vulture/budgets/suppression-owners clean · actionlint clean (it caught one real defect post-execution:
runner.tempin job-level env is an invalid context — fixed in f5a2b1b by moving to step-level env).What only this CI run can prove: the Linux and macOS cells end-to-end (Xvfb, Chrome identity per image, transport journey on macOS/Windows runners, that
macos-latestis truly ARM64 — asserted in-job by design).Human-gated after green (RELEASE-2 DoD): configure the repo ruleset requiring only
release-gate, run the two bite-proof experiments (deliberatequalityfailure + deliberate single-cell failure must each block merge), capture ruleset evidence. Until then the OS gate is not advertised.Merge order: after #42 → #43; this then auto-retargets. True merge commits, please.