RELEASE-5 W5: generated qualified release contract and tool evidence ledger - #54
Open
AminDhouib wants to merge 21 commits into
Open
AminDhouib wants to merge 21 commits into
AminDhouib wants to merge 21 commits into
Conversation
plan_RELEASE §2.5's `get_cookies` hard block clears via **option (a)**: a real
Chrome + real transport test now sets a cookie, retrieves it, and asserts its
value. W5 may claim 94 release-qualified tools.
Evidence node:
tests/test_e2e_transport_cookies.py::test_real_transport_cookie_round_trip
A dedicated collected node, deliberately: §2.5 rules that a *representative
journey* cannot carry a per-tool success claim, and §2.1 names the canonical
journey as exactly that — so folding these assertions into `_canonical_journey`
would have produced evidence W5 must reject. The machinery is still the one
harness (absolute installed launcher, isolated HOME/session root, fixture app,
stdio tools/call, bounded teardown); `stages="cookies"` selects a third declared
segment on top of it. No second journey mechanism, no second fixture.
The round trip, entirely through tools/call:
set_cookie -> get_cookies(urls=[url]) -> ASSERT VALUE -> get_cookies()
-> document.cookie cross-check -> clear_cookies -> assert gone
The value is unique per run so a stale cookie cannot forge the assertion, and
the document.cookie cross-check proves the cookie reached the browser rather
than get_cookies echoing what set_cookie was handed. Verified by mutation:
tampering with the value set makes the node fail on the value assertion.
set_cookie and clear_cookies are proved on the same real path (removal is
proved by re-reading, not by trusting the return value).
Why this was believed impossible, and what actually changed
----------------------------------------------------------
`get_cookies` was the sole `E2E_EXEMPT` name, on the grounds that it "hangs
against real Chrome ... and poisons the tab's CDP connection". That reason is
seam-specific, not product-wide. Measured on this base, same tool, same Chrome:
* in-process `.fn` seam — Network.getCookies AND getAllCookies both hang
(30s, no return), and the next call on that tab dies with a 10s CDP
timeout. The exemption's description of the symptom was accurate.
* real stdio transport + detached backend — both retrieval paths return,
document.cookie agrees, clear_cookies works, later calls are fine.
The transport path is the one users actually have, so the tool works; the E2E
suite simply could not reach it. `get_cookies` therefore moves from E2E_EXEMPT
to E2E_COVERED (covered by the transport node) and E2E_EXEMPT is now empty. The
seam hang is recorded at both call sites rather than erased, and is routed as a
finding — not fixed here: plan_RELEASE is zero-`src`.
Also noted for routing, deliberately NOT fixed: `get_cookies` is declared
`-> list[dict[str, Any]]` but returns nodriver `cdp.network.Cookie` dataclasses.
pydantic serializes those correctly, so the wire shape a user receives is right
and this is cosmetic; but fastmcp's `result.data` reconstructs them as an opaque
`[Root()]`, which is why this node asserts on `structured_content`.
Gates: ruff format+check clean; ty 76 diagnostics (baseline); vulture,
suppression owners, file budgets clean; unit lane 821 passed / 1 skipped.
W1's canonical journey re-verified green alongside the new node.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
W5 decides what a green gate may be CLAIMED to mean. Three pieces, one truth each. 1. `tools/release_evidence.py` -- the SOLE parser/generator for the closed `release-evidence/v1` schema. Every required job/matrix cell writes `release-evidence/<release_sha>/<job_id>/<matrix_cell>.json`; the `release-evidence` job re-reads all 30 of them through this one module and writes `.../release-gate/aggregate.json`. Each cell DECLARES what it owes (`expects_pytest`, `expects_chrome`, `expects_launched_chrome`), so `null` can never be a way to omit evidence: a browser cell with no Chrome identity, or a handshake-only macOS smoke cell that suddenly claims a launch, both fail. The aggregate fails closed on a missing/extra/duplicate child, a body that disagrees with its path, a stale release SHA, a foreign workflow run or attempt, a non-success terminal outcome, a runner with no GitHub-hosted image identity (self-hosted is outside the matrix), missing Chrome identity, an omitted pytest block, unsorted/duplicated node ids, a JUnit or artifact hash mismatch, an artifact never uploaded, a malformed/duplicate/absent MQ id -- and on any per-tool claim the run did not actually prove. 2. The gate wiring. Every required cell emits its record with `if: always()`, so a RED cell still writes a record saying it was red and the aggregate rejects it. `release-evidence` is a direct `needs:` edge of `release-gate` IN ADDITION to every child job it validates, never instead of one: the ledger can only make a green run red. 3. `tools/gen_release_contract.py` + root `RELEASE_CONTRACT.md`. The contract is generated and CI fails on drift (`--check` in `quality`, plus a unit test). Every number in it is derived: the served count from SECTION_TOOLS, the qualified count from the claim ledger -- each claim re-verified against the run's real records -- and the matrix from the ledger's required cells, which a workflow pin holds equal to release-gate.yml's matrix. What the contract says at this SHA, and why it is uncomfortable: plan_RELEASE 2.5 rules that a `.fn`-only call, an in-memory client, an exemption, a characterization, or THE REPRESENTATIVE JOURNEY cannot qualify a tool. The transport lane contains exactly one node -- the representative journey -- so no served tool has per-tool real-transport success evidence here. That is recorded as F-776 rather than smoothed over: the `get_cookies` hard block turns out to be the visible tip of a general gap, not a lone exception. The limitations register enumerates every open defect (E8-1..4, E7-1, E7-6, F-165, F-181, close-path flake, the unreproduced macOS `close_tab` observation, F-773, F-774, the F-775c/F-775d/`_replace_main_tab` residuals), every excluded surface (HTTP unauthenticated, the code-execution boundary, architectures and channels, native IME, live web), W7's exact public-surface exclusions, and every workstream that has NOT run -- including W14, so the contract makes no upgrade or rollback claim at all. Zero `src/` edits: tests, CI and docs only. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
The contract may only claim a tool whose cited node THIS run executes and passes -- the ledger rejects a claim it cannot verify. PR #52's dedicated real-transport node is that evidence for set_cookie/get_cookies/clear_cookies, so it is merged here rather than cited from a branch this gate does not run. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
The get_cookies hard block (plan_RELEASE §2.5) clears via option (a): PR #52's dedicated real-transport node sets a cookie, reads it back and asserts the exact value. Three claim rows follow from it — set_cookie, get_cookies, clear_cookies — because that one node asserts a distinct user outcome for each: the cookie really reaches the browser (document.cookie confirms it independently, so get_cookies cannot be echoing its own input), the set value returns from BOTH CDP retrieval paths against a per-run unique value, and removal is proved by re-reading rather than by a return value. The headline is therefore 3 of 94 release-qualified, not 93 or 94. Nothing else in the tree clears the §2.5 bar: the only other real-stdio evidence is the representative journey, which §2.5 disqualifies as per-tool evidence, and everything else is the in-process `.fn` seam or the in-memory client. That gap is F-776, and this number is its measure. - MQ-53 rewritten as option (a): "returns the exact value that was set", citing the node, with the bound stated — Linux/X64 + Windows/X64 only, macOS excluded under F-773. The old `[KNOWN-BUG: get_cookies_hang]` block is recorded as MEASURED to be the `.fn` seam (F-777), not reinterpreted away. - MQ-111 rewritten from an inventory requirement ("every tool has >=1 E2E test", which set equality can satisfy without behaviour) to "every served tool has a visible state in the contract", evidenced by the contract tests. - New register rows: F-777 (get_cookies hangs through the `.fn` seam and poisons the tab's CDP connection while succeeding over real stdio — a seam failure can misrepresent the served path in EITHER direction, so an exemption justified by seam behaviour may rest on a false premise), F-778 (declared return type vs. nodriver dataclasses; the WIRE shape is correct), and the observed install-smoke cold-spawn flake, which is why the contract refuses the flake-freedom claim §0.2 would otherwise need. - Ledger tests: the shipped claims are verified BOTH ways — clean when their node passes on the cells they name, red when it did not run. The structural tests use an empty claims document so they test the ledger, not the day's claims. No CI run has validated any of this yet: the ledger has no current-SHA evidence, so `release-evidence` will fail closed until this branch runs. That is the design working, not a defect to route around. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Human decision 2026-07-25: tag 2.0.0 now, at W5, rather than after W16 as plan_RELEASE originally stated. The cost was stated and accepted; the changelog therefore declares W6-W16 as NOT EVIDENCED rather than letting their absence read as a passing result. Bumped 1.2.0 -> 2.0.0 in pyproject.toml, the README install pins, and the RUNBOOK example. Left alone deliberately: - singleton.py's "<= 1.2.0" comments are historical statements about which released backends lack the version file. They are correct as written. - tools/package_verify.py and tests/ use 1.2.0 as an illustrative sample version; nothing binds the real package version (verified: no importlib.metadata/__version__ assertion anywhere in src/ or tests/). - smoke_mcp.py is gitignored (untracked local helper); its version assertion was updated in the working copy only. The major bump is required regardless of scope: STEALTH_MCP_SESSION_STORAGE_CAP_GB -> STEALTH_MCP_BROWSER_SESSION_STORAGE_CAP_GB and --session-cap-gb -> --browser-session-cap-gb ship with NO back-compat alias, so the old names lapse silently on upgrade. That is the lead item in the changelog. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
…PPING one
Two inputs land here.
1. The `get_cookies` hard block cleared via plan_RELEASE 2.5 option (a). The
merged node sets a cookie over real stdio against real Chrome, reads back its
EXACT value from both CDP retrieval paths, cross-checks `document.cookie`, and
proves removal by RE-READING. It is a dedicated collected node, never the
representative journey, so 2.5 accepts it -- and it qualifies three tools on
the one path: set_cookie, get_cookies, clear_cookies. Bound stated in the
contract, not implied: the transport lane is Linux/X64 + Windows/X64, so those
rows are qualified on exactly TWO cells, never three (F-773).
The claim rows go through the ledger like any other: the `release-evidence`
job re-checks that this run executed and passed that node on both cells, and a
local test refuses a claim citing a node this tree does not define.
2. The human is tagging now rather than after W16, so this document is what
users read before installing. Rewritten accordingly:
* it names the version it ships (read from pyproject.toml -- a tagged run
already fails on a tag/version disagreement, so the two cannot diverge);
* the 1.x BREAKING CHANGE is above the tables, not buried in a register:
STEALTH_MCP_SESSION_STORAGE_CAP_GB and --session-cap-gb were renamed with NO
back-compat alias. The env var is the dangerous one -- nothing errors, the
configured storage cap silently stops applying;
* it refuses the blind-push property outright. plan_RELEASE 0.2 rests it on
manual-QA parity, flake-freedom and mutation-informed strength; none of the
three exists, so the contract says the gate is strong on what it covers and
is not a substitute for a human release pass;
* every workstream that produced nothing reads NOT EVIDENCED with "the reader
may not infer that it was checked" -- not "planned", which reads as a
roadmap for a document that is shipping.
Three findings recorded rather than smoothed over:
* F-777 -- `get_cookies` through the in-process `.fn` seam hangs AND poisons the
tab's CDP connection for the next call. Same tool, same Chrome, fine over real
stdio: the blast radius is the harness seam, not the user's path. That is why
the E2E exemption's stated reason was wrong, and why the tool's coverage now
lives in the transport lane.
* F-778 -- `get_cookies` is declared `-> list[dict[str, Any]]` but returns
nodriver Cookie dataclasses. The wire shape is correct (pydantic serializes
them); only fastmcp's `.data` reconstruction is opaque. Cosmetic, recorded.
* install-smoke (sdist Linux/X64) cold-spawn flake -- one first-attempt failure
on a Chrome cold spawn, passed on re-run, warmup retry did not absorb it. A
known flake in a required cell belongs in a shipping contract; W8 owns the
disposition, so the contract asserts no flake-freedom.
F-776 narrows from "no tool has per-tool transport evidence" to "only the cookie
tools do" -- the gap is real for every other served tool and the register says so.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
A drift check that only says "the contract is stale" sends the reader to regenerate blindly -- which is exactly how a deliberate change gets rubber-stamped instead of reviewed. The failure now prints the first differing lines, so whoever hits it (the 2.0.0 version bump will, since the contract names the version it ships) can see WHAT changed before they accept it. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
W5's contract generator resolved to 94 served / 3 release-qualified, not 94 qualified. plan_RELEASE 2.5 disqualifies exactly the evidence the other 91 have: a `.fn`-only call and the representative journey cannot satisfy a transport or cross-OS success claim. Verified independently: `pytest -m transport --collect-only` yields ONE node on this base. My earlier "the contract can claim all 94" was wrong - I carried forward the get_cookies hard-block framing instead of re-deriving the per-tool rule. The changelog was silent on this. Stating it, and stating it in BOTH directions: the 91 are covered end-to-end against real Chrome (set-equality tripwire, 94 covered / 0 exempt), they simply are not verified over the wire. Omitting it would overclaim; reporting a bare "3 qualified" would read as "91 untested", which is equally false. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
…e sdist Found while dry-running the publish path for a v2.0.0 tag. The sdist was 15 MB against a 192 KB wheel: 12.8 MB assets/ (demo media) + 2.8 MB audit/ (the campaign's internal plans and findings). W3's install-smoke installs the sdist on three runner cells, so the weight was paid on every gate run as well as by anyone building from source. Verified safe before changing anything: package_verify's `_check_sdist_members` requires only pyproject.toml + README.md at the sdist root, and `_check_package_data` requires the embedded/js files, which live under src/. Neither excluded directory is structurally required. tests/ is kept deliberately so a downstream packager can still run the suite from the sdist. Measured locally, not assumed: sdist 15 MB -> 592 KB (25x) wheel 192 KB -> 192 KB (unchanged) package_verify verify --expect-version v2.0.0 -> OK, before and after tests/test_package_verify.py + test_release_workflows.py -> 37 passed Also confirms the tag/version agreement gate accepts the `v2.0.0` form, so that failure mode cannot surface late in the tag run. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
…" reading as "untested" The first CI run turned 24 cells red, and both causes were real. 1. Node ids. Every pytest cell wrote ids like `tests.test_backend_logging.TestBootstrapBackendProcessLogging::test_creates_backend_log_file` -- a dotted module path, which is NOT a node id and cannot be resolved against `pytest --collect-only`. The validator caught it on the first run, which is exactly what it is for. The fix is in the PRODUCER, not the check. This pytest configuration writes JUnit with `classname`/`name` only -- no `file` attribute -- so a node id could at best be guessed from a dotted string. The gate now passes `-o junit_family=xunit1`, which writes the source `file`, making the id exact (parametrised cases included, since the parameters live in `name`). The check stays strict: a dotted id is now pinned as a REJECTION case, and a real pytest subprocess pins the producer contract, so a future pytest that drops the `file` attribute fails in the unit suite instead of silently producing ids nobody can verify. A workflow pin asserts every pytest lane passes the flag. 2. `runner.image_os` rejected `win25-vs2026`. Hosted image ids carry hyphens and dots; the pattern was wrong, not the rule. What must stay rejected -- an ABSENT image identity, the self-hosted signal -- still is, and still has its test. The corrected pattern gets one too. Then the other half, which is a correction in the opposite direction: "3 qualified / 91 served-unqualified" would be read as "91 untested tools". That is false. Those tools are driven against REAL Chrome by the E2E suite, with a set-equality tripwire keeping the coverage complete (94 covered, 0 exempt). What they lack is a per-tool assertion over the stdio transport. So the contract now says that in plain words immediately under the headline number, every served-unqualified row states the evidence that EXISTS rather than repeating its label, and a genuinely evidence-free tool would get its own visibly weaker row instead of hiding in the same bucket (none is in that state). F-776 carries the same two-directional framing, and tests pin both halves. F-777 (the `.fn`-seam hang plus CDP connection poisoning) and F-778 (the declared-type mismatch, wire shape correct) are now written up as findings and cross-linked from the register. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
The ceiling paragraph and the limitations row both described this flake as one cosmetic install-smoke cell that passed on re-run. Run 30168575218 says otherwise: the same "Failed to connect to browser" cold-start race took out `transport (Linux/X64)` AND `install-smoke (wheel Linux/X64)` in one run -- and `transport (Linux/X64)` is one of the two cells that carry every qualified stdio claim in this contract. The cookie test in that same job spawned Chrome successfully seconds later, so it is a race, not a defect; but a reader deciding how much a qualified claim is worth needs to know the flake can hit the evidence the claim rests on. Understating it was the kind of laundering this whole workstream exists to prevent. Also state the one-attempt property the ledger enforces: every record is bound to a single run_id + run_attempt, so re-running only the failed jobs can never certify -- the surviving records belong to the earlier attempt and are refused on identity. That is the fail-closed rule working as designed, but it is operationally surprising enough that the contract should say it out loud rather than let someone read the resulting red as a second failure. The flake-freedom test now also pins that the transport cell is named, so a future edit cannot quietly shrink this back to a cosmetic-cell story. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
The contract is generated FROM pyproject's version, so a version bump that lands without regenerating it leaves the contract describing a version that no longer exists. W5's completion note called this out explicitly: regenerate in the same commit as any bump. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
…scribes The contract was generated at 1.2.0 while its own body documents a BREAKING change from 1.x: STEALTH_MCP_SESSION_STORAGE_CAP_GB and --session-cap-gb were renamed with no back-compat alias (verified in settings.py:57-58 and cli.py:461). A 1.2.0 release cannot carry a break from 1.x, and the env rename is the dangerous half - nothing errors, the configured storage cap silently returns to default. W5's completion note required regenerating the contract in the same commit as any version bump. This merges the release/v2.0.0 line so the generated header, the tag assertion, and the breaking-change section all agree. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Mechanical output of `gen_release_contract.py --write` after the 2.0.0 merge. The header now reads 2.0.0, so it agrees with pyproject, with the tag assertion, and with its own "Breaking change from 1.x" section. `--check` is clean. The drift tripwire earned its keep here: it failed on the real mismatch rather than letting a contract ship that described a version it was not generated for. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Empty commit, no tree change. d13997e's only delta vs the fully-green 5c2505a run is version metadata + sdist excludes (already verified by d13997e's own six green install-smoke cells); the integration (macOS/ARM64) 'Event loop is closed' failure is therefore suspected environmental. A second run on identical code arbitrates flake vs deterministic before any code gets touched. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
The contract regeneration commit (d13997e) went red on integration (macOS/ARM64) with `Event loop is closed` at teardown, taking release-evidence and the release-gate aggregate down with it. Re-running the gate against a BYTE-IDENTICAL tree -- an empty commit built from d13997e's own tree object -- returned 32/32 success, including that cell. Same code, same workflow, same runner image, two different conclusions. The tree is exonerated; the gate is not. The mechanism stays undiagnosed on purpose rather than by guess: job logs require admin rights, so only check-run annotations were readable, and they carry the message without a traceback. The finding says so instead of inventing a cause. This is the SECOND distinct gate flake on record, and unlike the Linux cold-spawn one it reddens the aggregate check itself -- the very check a ruleset is meant to require. A required check that intermittently fails for reasons unrelated to the change trains reviewers to re-run until green, which is indistinguishable from training them to ignore it. So the contract's flake-freedom disclaimer now names both flakes rather than describing the gate as having one bad cell. Routed to W8 (flake quarantine), whose acceptance criterion should be "identical tree, different conclusion" -- not a retry budget. Adding a retry here would hide a teardown bug behind exactly the second-way-to- do-something defect the repo conventions forbid. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
The 2.0.0 bump moved pyproject.toml's version but left uv.lock still recording `version = "1.2.0"` for the editable root package. Any `uv sync` regenerates it, so the file showed up dirty in a clean worktree -- which is how it was found. Left alone this is the same class of defect the release campaign keeps hitting: two files that disagree about what this thing IS, where only one of them is the one people read. The lockfile is what actually governs an installed environment, so a stale version there is a metadata lie with a real consumer, not cosmetic drift. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
…n (F-780) #43, #45 and #47 are red on the legacy `Browser Integration Tests (Chrome + Xvfb)` check while main and #42 are green on the same check. Read at face value that says FIX-B broke browser integration. It did not, and the reason is provable without reading a single CI log. test.yml runs `pytest -m integration --timeout=120`. The harness it drives declares INIT=60, LIST=130, SPAWN=120, WARMUP=150 with 4 attempts -- under a comment saying "the pytest --timeout is the outer net". Two of those single-step bounds are LARGER than the whole job's per-test budget, and BACKEND_READY_TIMEOUT is exactly equal to it. A 120s outer net cannot contain a 150s inner step. The job is structurally incapable of finishing that test; no timing luck is involved. It only started failing at FIX-B because W1 landed the transport journey already xfail'd (B1 was open), and an xfail costs nothing on the clock. FIX-B C2 correctly removed the marker once B1 was fixed, so from that commit the test actually runs -- and immediately hits the wall. The controlled comparison is #46: it contains FIX-B transitively, runs the SAME `-m integration` selection on the SAME OS under the release gate's 180s budget, and is green 23/23. Deliberately NOT fixed. W2 deletes this job, so patching a doomed file on three in-review branches is churn that also mutates PRs under human review. Recorded instead, because the trap is expensive: the merge queue will show red at #43 -> #45 -> #47 until W2 lands, and someone will otherwise go hunting for a regression that was never there. Not added to RELEASE_CONTRACT.md on purpose -- the contract describes what a user receives, not our merge plumbing. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
…(2/6) I wrote F-779 as "observed once then not reproduced". That was wrong within hours, and the correction matters more than the original entry. 7c65374 adds exactly one file: a markdown finding. No code, no workflow, no test, no dependency. integration (macOS/ARM64) failed on it with the byte-identical signature -- `Event loop is closed` twice at teardown. A markdown file cannot break a macOS integration test, so together with the earlier identical-tree re-run the code is now exonerated twice by two independent methods. Measured across six consecutive runs on this line the cell fails 2/6 (~33%), successes bracketing each failure. Since the aggregate demands every edge green, one cell at 33% puts the HEADLINE release-gate check red about one run in three no matter how healthy the other 31 jobs are. That is the honest headline of this release, so the contract now says it in those words rather than burying it in a limitations row: while F-779 is open, a green check is evidence about that run only, and a red check is not by itself evidence about your change. That ambiguity is precisely what a release gate exists to remove, so the "green => blindly pushable" goal is not met -- not because the software is bad, but because the instrument is unreliable. Correctness of 2.0.0 is unaffected; the tree is green at 6a8fa79 and a97c970 and the failures are provably code-independent. Whether to tag anyway is a human call. What is no longer available is calling the gate trustworthy while this is open. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
…t it
Reading the teardown path gives a specific candidate, so the finding now
points at code rather than leaving the next person with "macOS is flaky".
close_instance Phase 3 wraps asyncio.to_thread in asyncio.wait_for with
CLOSE_KILL_TIMEOUT (5s). wait_for cancels the AWAITABLE; it cannot cancel
the worker THREAD -- the code's own comment says so ("worker thread
continues in background"). So on timeout a thread survives, still holding
`browser`, after close_instance returned.
Two ways that thread yields exactly `Event loop is closed` once the loop
is gone: the executor resolving its future via call_soon_threadsafe, and
browser._process.terminate() at :238 -- nodriver's _process is an ASYNCIO
subprocess bound to that loop, so terminate()/kill() from the thread hit
_check_closed(). The second path failing twice through the retry loop is
a plausible reading of why the annotation shows the message twice.
It fits the fingerprint: intermittent (only when kill exceeds 5s),
teardown-only, code-independent (hence the markdown-only commit), and
macOS-leaning, which is already the anomalous cell per F-773.
Explicitly labelled a HYPOTHESIS with zero direct confirmation -- no Mac
was available and job logs need admin rights, so none of this was
observed. The finding says how to falsify it in one step: check whether
the traceback originates in a ThreadPoolExecutor thread. If it does not,
the section should be deleted, not defended.
No fix here. plan_RELEASE forbids src/ edits, and the honest fix is
structural (a thread that cannot be cancelled is abandoned holding
loop-bound objects) rather than widening the timeout, which would only
move the race. Routed to a FIX plan.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
The rate has now read 2/6, 2/7 and 2/8 -- not because anything changed, but because every commit documenting it adds a sample to its own denominator. That is a genuinely self-invalidating measurement, and the honest response is to name the property rather than keep re-editing a number that is structurally always slightly behind. So the finding and the contract now lead with what the drift cannot touch: the macOS/ARM64 integration cell fails intermittently at an order of roughly one run in four, code-independently, and each failure takes the release-gate aggregate down with it. The exact fraction is recorded with its as-of commit and explicitly marked live. Also fixed two internal inconsistencies left by the earlier correction (a "fails a third of the time" and a "red one run in three" that the newer sample had already outdated), and noted that one-in-four is a FLOOR for the aggregate, not the whole story -- the Linux cold-spawn flake can redden the same check independently. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
RELEASE-5 (W5) of
plan_RELEASE— the qualified release contract, the limitationsregister, and the ONE generated evidence source. Base:
audit/release-integration(PR #51, 31/31 green). Zero
src/edits.This PR also merges PR #52 (
audit/release-get-cookies,eaa574e): the contract mayonly claim a tool whose cited node this run executes and passes, so the node had to be in
this tree rather than cited from a branch this gate does not run.
What landed
tools/release_evidence.py— the sole parser/generator forrelease-evidence/v1.30 required job/matrix cells each write
release-evidence/<release_sha>/<job_id>/<matrix_cell>.json; the newrelease-evidencejob re-reads all of them through this one module and writes
release-evidence/<release_sha>/release-gate/aggregate.json.Every cell declares what it owes (
expects_pytest,expects_chrome,expects_launched_chrome), sonullcan never be a way to omit evidence: a browser cellwith no Chrome identity fails, and a handshake-only macOS smoke cell that suddenly claims a
launch fails too.
Fail-closed rejection paths, each with a negative test (
tests/test_release_evidence.py,55 tests): missing / extra / duplicate child · body-vs-path disagreement · stale release SHA
· foreign
run_id/run_attempt· non-success terminal outcome · invalid outcome enum ·unknown or missing field · malformed SHA / run id / cell · runner with no GitHub-hosted image
identity (self-hosted is outside the matrix) · missing Chrome identity on a browser cell ·
launch claimed by a non-launching cell · pytest block omitted where owed or invented where
not · unsorted or duplicated node ids · JUnit hash mismatch · artifact hash mismatch ·
artifact never uploaded · unknown artifact kind · malformed / duplicated / absent MQ id ·
claim whose node never ran · claim whose node was skipped · claim citing the
representative journey · stdio claim evidenced by a job that does not run the stdio
lane · claim naming an undeclared cell · claim with no cells.
The gate wiring. Every required cell emits with
if: always(), so a red cell stillwrites a record saying it was red — and the aggregate rejects it.
release-evidenceis adirect
needs:edge ofrelease-gatein addition to every child it validates, neverinstead of one. The ledger can only make a green run red.
actionlintclean.tools/gen_release_contract.py+ rootRELEASE_CONTRACT.md— generated, with--checkin
qualityand a unit test that prints the diff on drift.The number: 94 served, 3 release-qualified
set_cookie/get_cookies/clear_cookies, ontests/test_e2e_transport_cookies.py::test_real_transport_cookie_round_trip, qualified onLinux/X64 + Windows/X64 — two cells, never three (macOS transport is excluded under
F-773). The count is derived from the claim ledger and re-verified against the run's records;
nothing about it is typed.
The other 91 tools are
served-unqualified. That is F-776, opened by this PR: §2.5 rulesout a
.fn-only call, an in-memory client, an exemption, a characterization, and therepresentative journey as per-tool evidence — and the transport lane's only other node is
that journey. The
get_cookieshard block turned out to be the visible tip of a general gap.Also recorded
get_cookiesthrough the in-process.fnseam hangs and poisons the tab'sCDP connection for the next call. Same tool, same Chrome, correct over real stdio: the
blast radius is the harness seam, not the user's path.
get_cookiesdeclared-> list[dict[str, Any]], returns nodriverCookiedataclasses; wire shape correct, only fastmcp's
.datareconstruction is opaque.install-smoke (sdist Linux/X64)cold-spawn flake — one first-attempt failure, passedon re-run, warmup retry did not absorb it. The contract therefore asserts no
flake-freedom.
Written as a shipping 2.0.0 contract
pyproject.toml; a tagged run already fails on atag/version disagreement);
STEALTH_MCP_SESSION_STORAGE_CAP_GBand--session-cap-gbrenamed with no back-compat alias — the env var fails silently;strength; none exist);
Human actions
uv run python tools/gen_release_contract.py --write). The drift test will say so, with a diff.Gates: ruff format/check clean · suppression owners clean · file budgets clean, no cap padded
·
ty76 diagnostics (baseline) · full unit suite 910 passed / 1 skipped, no new skips ·actionlintclean.