From aa8b5df172a8dda73d9ab650775109d8c5f1b93a Mon Sep 17 00:00:00 2001 From: Hermes Agent Date: Mon, 14 Sep 2026 11:16:01 -0500 Subject: [PATCH 1/2] fix(ci): quarantine three merge-group flaky tests (#2740) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Quarantine the three auto-detected flaky test paths from merge-group CI run 34726206593 (filed by flake-detection workflow run 34728138233, issue #1782) with OWNER/EXPIRY metadata per issue #2477: - tests/unit/services/evidence-summary-adversarial.test.ts — general ledger (scripts/ci/quarantined-tests.txt). Flake: unit (ubuntu-latest, 2) shard, attempts 1+2 failed, passed on retry 2. The 'should not block on slow event handlers' case registers a real 100ms setTimeout handler and asserts expect(duration).toBeLessThan(500) at tests/unit/services/evidence-summary-adversarial.test.ts:882 — under merge-group runner CPU contention the publish exceeds the 500ms budget. No ubuntu-specific ledger exists, so the general ledger is used, matching the #2368 dispatch-lanes precedent. - tests/unit/utils/bun-compat-exit-first-2530.test.ts — macOS ledger (scripts/ci/quarantined-tests-macos.txt). Flake: unit (macos-latest, 4) shard, passed on retry 1. Shells out to bun build + node probes under hard 5s timeouts (PROBE_TIMEOUT_MS at line 9); a runner stall >5s on cold build or CPU contention trips the probe timeout. - tests/unit/commands/promote-registration.test.ts — macOS ledger. Flake: unit (macos-latest, 6) shard, passed on retry 1. beforeEach mutates process.env.HOME/LOCALAPPDATA/XDG_DATA_HOME to redirect the hive knowledge path into a fresh mkdtempSync temp dir (cleaned in afterEach) — the classic env-mutation + temp-dir teardown surface. All three files pass locally (62 pass / 0 fail in 4.51s), confirming environment-sensitivity rather than a logic bug; the ci.yml retry loop discards attempt-1 output when a retry passes, so no failing assertion text exists to drive a root-cause fix. The unit (windows-latest, 4) failure in the same run was close-active-state-unlink-retry.test.ts (unrelated, not flagged). Verification: - ci.yml grep|sort|comm consumer pipeline: all 3 paths present in discovery (3501 files), all 3 removed from gated-tests; no same-basename siblings affected. - scripts/check-invariants.ts Check 7: all entries carry OWNER/EXPIRY (EXPIRY 2026-10-14, 30 days out). - bun run test:unit:ci : 1 quarantined on Linux (general ledger), 2 run (macOS-scoped ledger) — matches platform semantics. - tests/unit/scripts/ci/ci-yml-integration.test.ts: 31/31 pass. - tests/unit/scripts/ci/ci-yml-windows-quarantine.test.ts: 4/4 pass. - tests/unit/scripts/check-quarantine-metadata.test.ts + ci-coverage- sharding.test.ts: 37/37 pass. - bun run check:pending-fragment: OK (fragment present, all refs verified). Refs: #2740 (this issue), #1782 (flake-detection workflow), #2477 (OWNER/EXPIRY grammar), #1908 (general-ledger retirement precedent), #2368 (general-ledger ubuntu precedent), #2738 (macOS ledger precedent). --- ...hree-merge-group-flaky-tests-quarantine.md | 79 +++++++++++++++++++ scripts/ci/quarantined-tests-macos.txt | 41 ++++++++++ scripts/ci/quarantined-tests.txt | 24 ++++++ 3 files changed, 144 insertions(+) create mode 100644 docs/releases/pending/issue-2740-three-merge-group-flaky-tests-quarantine.md diff --git a/docs/releases/pending/issue-2740-three-merge-group-flaky-tests-quarantine.md b/docs/releases/pending/issue-2740-three-merge-group-flaky-tests-quarantine.md new file mode 100644 index 000000000..481f4ae61 --- /dev/null +++ b/docs/releases/pending/issue-2740-three-merge-group-flaky-tests-quarantine.md @@ -0,0 +1,79 @@ +# Quarantine three merge-group flaky tests (issue #2740) + +## What changed + +- Appended three new entries to the CI quarantine ledgers, each carrying the + structured `# OWNER:` / `# EXPIRY:` metadata block required by + `scripts/check-invariants.ts` Check 7 (issue #2477): + - `tests/unit/services/evidence-summary-adversarial.test.ts` → general + ledger, `scripts/ci/quarantined-tests.txt` (ubuntu-latest flake; no + per-OS ledger exists for ubuntu, matching the #2368 dispatch-lanes + precedent) + - `tests/unit/utils/bun-compat-exit-first-2530.test.ts` → macOS ledger, + `scripts/ci/quarantined-tests-macos.txt` + - `tests/unit/commands/promote-registration.test.ts` → macOS ledger, + `scripts/ci/quarantined-tests-macos.txt` +- No source, test, or hook code changed. The two ledger files are the only + repository files modified, plus this pending release fragment. + +## Why + +Issue #2740 was auto-filed by the `flake-detection` workflow (issue #1782, +`scripts/ci/detect-and-quarantine-flakes.sh`) after a merge-group CI run +reported all three paths as flaky candidates (each with a `Passed on retry` +annotation, so the ci.yml retry loop discarded the attempt-1 output — no +assertion text exists to drive a root-cause fix). + +Triggering merge-group run 34726206593 (filed by flake-detection run +34728138233 at 2026-09-13T00:30Z), with the flake annotation located in each +shard's log: + +- `tests/unit/services/evidence-summary-adversarial.test.ts` — `unit + (ubuntu-latest, 2)`; attempts 1 and 2 both failed, passed on retry 2. The + "should not block on slow event handlers" case registers a real 100 ms + `setTimeout` handler and asserts + `expect(duration).toBeLessThan(500)` at + tests/unit/services/evidence-summary-adversarial.test.ts:882 — under + merge-group runner CPU contention the publish can exceed the 500 ms budget. + Sibling ubuntu/macos/windows shards green. +- `tests/unit/utils/bun-compat-exit-first-2530.test.ts` — `unit + (macos-latest, 4)`; passed on retry 1. Most tests shell out to `bun build` + plus a `node` probe under hard 5 s timeouts + (`PROBE_TIMEOUT_MS` at + tests/unit/utils/bun-compat-exit-first-2530.test.ts:9) — a macos-latest + runner stall >5 s trips the probe timeout. Sibling shards green. +- `tests/unit/commands/promote-registration.test.ts` — `unit + (macos-latest, 6)`; passed on retry 1. `beforeEach` mutates + `process.env.HOME` / `LOCALAPPDATA` / `XDG_DATA_HOME` to redirect the hive + knowledge path into a fresh `mkdtempSync` temp dir (cleaned in `afterEach`) + — the classic env-mutation + temp-dir teardown surface (#1782 class). + Sibling shards green. + +All three files pass locally on this checkout (62 pass / 0 fail across the +three files in 4.51 s), confirming the flakes are environment-sensitive +(merge-group runner pressure) rather than logic bugs. The `unit +(windows-latest, 4)` failure in the same run was `close-active-state-unlink- +retry.test.ts` — unrelated (already root-fixed surface, different file, and +the detect job did not flag it). + +## Migration steps + +None. Quarantine is a CI-gating data change: the ci.yml unit-shard discovery +pipeline (`grep | sort | comm`) now excludes these three paths from the gated +test set on the relevant OS (general ledger applies to all OSes, macOS ledger +to macos-latest only). + +## Known caveats + +- Entries carry EXPIRY 2026-10-14 (30 days out; inside the 14-day grace + window Check 7 only warns). They must be root-fixed or renewed before the + expiry closes; the recommended root fixes are noted in each entry's + `# EXPIRY` criterion line. +- Ledger placement is per-OS evidence-based: the evidence-summary flake was + single-detection on ubuntu-latest so it landed in the general ledger (no + ubuntu-specific ledger exists); the two macOS flakes landed in the macOS + ledger. If any file later flakes on a second OS, it should be split or + moved accordingly. +- These are the first macOS-ledger entries added alongside the #2738 signal + test; both ledgers are independent of the windows ledger (STATUS: 2 + entries) and the general ledger (now 1 active entry). diff --git a/scripts/ci/quarantined-tests-macos.txt b/scripts/ci/quarantined-tests-macos.txt index 0c9ff728c..4a0894f35 100644 --- a/scripts/ci/quarantined-tests-macos.txt +++ b/scripts/ci/quarantined-tests-macos.txt @@ -21,3 +21,44 @@ # EXPIRY: 2026-09-26 — remove when the fixture/harness signal handling is # fixed and the test passes on macos-latest. tests/unit/scripts/ci/repository-validation-real-process-2675.test.ts +# +# tests/unit/utils/bun-compat-exit-first-2530.test.ts (issue #2740) +# --------------------------------------------------------------------------- +# Regression coverage for #2530: verifies the bunSpawn Node fallback consumes +# stdout/stderr exit-first (empty/small/large/nonzero dual-stream matrix, +# bounded-output overflow, timeout termination) across both native Bun and the +# plain-Node fallback, plus the public isCleanWorktree path under node +# --input-type=module. Every non-matrix test shell outs to real processes: +# `bun build` of src/utils/bun-compat.ts and a `node` probe, both under hard +# 5s timeouts (PROBE_TIMEOUT_MS at tests/unit/utils/bun-compat-exit-first-2530.test.ts:9). +# A macos-latest merge-group runner stalled by >5s on cold build/install or +# CPU contention trips the probe timeout, which surfaces as attempt-1 failure +# with no product defect involved. Pre-existing, unrelated to any specific PR. +# Quarantined per issue #2740; tracked under #1782 (test-stability sprint). +# OWNER: zaxbysauce — issue #2740 (macos-latest merge-group retry-flake, +# run 34726206593 `unit (macos-latest, 4)`; siblings green) +# EXPIRY: 2026-10-14 — root-fix the probe timeout sensitivity (e.g. raise +# PROBE_TIMEOUT_MS or pre-warm the build) or renew with an updated +# criterion. Inside the 14-day grace window Check 7 only warns. +tests/unit/utils/bun-compat-exit-first-2530.test.ts +# +# tests/unit/commands/promote-registration.test.ts (issue #2740) +# --------------------------------------------------------------------------- +# Unit tests for the `/swarm promote` command registration and behavior +# (command import/export, switch-case registration, help text, direct-text +# mode, --category/--from-swarm/--force/--reason parsing, and the #1821 A3 +# actionability-floor gate). beforeEach mutates process.env.HOME / +# LOCALAPPDATA / XDG_DATA_HOME to redirect the hive knowledge path into a +# fresh mkdtempSync temp dir, and afterEach restores it and rmSync's the dir — +# the classic env-mutation + temp-dir teardown surface (#1782 class) plus a +# real filesystem write to the hive path per test. Attempt-1 failure on +# macos-latest merge-group runner with immediate retry pass is consistent +# with cold-file / env-read sensitivity, not a product regression. +# Pre-existing, unrelated to any specific PR. +# Quarantined per issue #2740; tracked under #1782 (test-stability sprint). +# OWNER: zaxbysauce — issue #2740 (macos-latest merge-group retry-flake, +# run 34726206593 `unit (macos-latest, 6)`; siblings green) +# EXPIRY: 2026-10-14 — root-fix the per-test env mutation / temp-dir teardown +# or renew with an updated criterion. Inside the 14-day grace window +# Check 7 only warns. +tests/unit/commands/promote-registration.test.ts diff --git a/scripts/ci/quarantined-tests.txt b/scripts/ci/quarantined-tests.txt index 998a02b2a..62bb41d1a 100644 --- a/scripts/ci/quarantined-tests.txt +++ b/scripts/ci/quarantined-tests.txt @@ -12,3 +12,27 @@ # the grace window it only warns). # # No active global test quarantines. Issue #1908 retired the tracked backlog. +# +# tests/unit/services/evidence-summary-adversarial.test.ts (issue #2740) +# --------------------------------------------------------------------------- +# Adversarial security tests for the Evidence Summary Pipeline (Task 5.8): +# malformed evidence payloads, blocker spoofing, artifact path abuse, and +# event spam — exercised through buildEvidenceSummary / +# createEvidenceSummaryIntegration with mocked loadEvidence / loadPlanJsonOnly. +# Core timing hazard: the "should not block on slow event handlers" case +# registers a real 100ms setTimeout handler and asserts +# expect(duration).toBeLessThan(500) at +# tests/unit/services/evidence-summary-adversarial.test.ts:882 — under +# merge-group runner CPU contention (6 unit shards co-scheduled with coverage +# and smoke jobs) the publish can exceed the 500ms budget and fail attempt 1, +# then pass on retry when the runner has settled. The temp dir per test is +# also mkdtempSync/rmSync'd. Pre-existing, unrelated to any specific PR. +# Quarantined per issue #2740; tracked under #1782 (test-stability sprint). +# OWNER: zaxbysauce — issue #2740 (ubuntu-latest merge-group retry-flake, +# run 34726206593 `unit (ubuntu-latest, 2)`; attempts 1+2 failed, passed on +# retry 2; no per-OS ledger exists for ubuntu so this uses the general one, +# matching the #2368 dispatch-lanes precedent) +# EXPIRY: 2026-10-14 — root-fix the 500ms wall-clock budget (e.g. freeze the +# clock or widen the margin) or renew with an updated criterion. Inside the +# 14-day grace window Check 7 only warns. +tests/unit/services/evidence-summary-adversarial.test.ts From 28f8cb97cba5329f55820f37d3711f8c61526a7d Mon Sep 17 00:00:00 2001 From: Hermes Agent Date: Mon, 14 Sep 2026 11:49:00 -0500 Subject: [PATCH 2/2] test(ci): pin issue #2740 quarantine ledger entries Add tests/unit/scripts/ci/ci-yml-quarantine-2740.test.ts with 9 regression tests (3 per quarantined path, mirroring the #2761 / #2368 / #2185 precedents) covering the three entries the previous commit added for the flaky-test candidates the merge-group flake-detection workflow (#1782) auto-filed in issue #2740: - tests/unit/services/evidence-summary-adversarial.test.ts -> general ledger - tests/unit/utils/bun-compat-exit-first-2530.test.ts -> macOS ledger - tests/unit/commands/promote-registration.test.ts -> macOS ledger Each path gets three guards: 1. active entry in its expected ledger (active-entry extraction mirrors the ci.yml grep -vE pipeline exactly; CRLF normalized) 2. scoped to that ledger only - no duplicate in the general/macos/windows/integration ledgers (cross-ledger duplicates would falsely imply per-OS evidence) 3. path exists on disk and is reachable by the ci.yml unit find chain (a typo would silently no-op the quarantine) RED/GREEN verification: - With entries in place: 9/9 pass (~250 ms). - With entries removed (sed deletion): 3/9 fail (the active-entry tests), confirming the tests drive the ledger fix rather than pass vacuously. - Full fast suite: ci-yml-integration + ci-yml-windows-quarantine + new file 44/44 pass; check-quarantine-metadata + ci-coverage-sharding 37/37 pass; the three quarantined files themselves run green locally (62 pass, clean TMPDIR). - bun x tsc --noEmit: clean. biome check --write applied. check:invariants (incl. Check 7 OWNER/EXPIRY): all pass. Also corrects the pending release fragment's stale claim that no test code changed. Refs: #2740, #1782 (flake-detection workflow), #2477 (OWNER/EXPIRY grammar), #2761/#2368/#2185 (consumer-side quarantine-pin test precedents). --- ...hree-merge-group-flaky-tests-quarantine.md | 6 +- .../scripts/ci/ci-yml-quarantine-2740.test.ts | 109 ++++++++++++++++++ 2 files changed, 113 insertions(+), 2 deletions(-) create mode 100644 tests/unit/scripts/ci/ci-yml-quarantine-2740.test.ts diff --git a/docs/releases/pending/issue-2740-three-merge-group-flaky-tests-quarantine.md b/docs/releases/pending/issue-2740-three-merge-group-flaky-tests-quarantine.md index 481f4ae61..ba8aebd09 100644 --- a/docs/releases/pending/issue-2740-three-merge-group-flaky-tests-quarantine.md +++ b/docs/releases/pending/issue-2740-three-merge-group-flaky-tests-quarantine.md @@ -13,8 +13,10 @@ `scripts/ci/quarantined-tests-macos.txt` - `tests/unit/commands/promote-registration.test.ts` → macOS ledger, `scripts/ci/quarantined-tests-macos.txt` -- No source, test, or hook code changed. The two ledger files are the only - repository files modified, plus this pending release fragment. +- No source or hook code changed. The two ledger files are the only + repository files modified, plus a consumer-side regression test + (`tests/unit/scripts/ci/ci-yml-quarantine-2740.test.ts`) that pins the three + new ledger entries, and this pending release fragment. ## Why diff --git a/tests/unit/scripts/ci/ci-yml-quarantine-2740.test.ts b/tests/unit/scripts/ci/ci-yml-quarantine-2740.test.ts new file mode 100644 index 000000000..cba7338e6 --- /dev/null +++ b/tests/unit/scripts/ci/ci-yml-quarantine-2740.test.ts @@ -0,0 +1,109 @@ +import { describe, expect, test } from 'bun:test'; +import { existsSync, readFileSync } from 'node:fs'; +import { join } from 'node:path'; + +// Repo root is four levels up from tests/unit/scripts/ci/. +const REPO_ROOT = join(import.meta.dir, '../../../..'); +const GENERAL_LEDGER_PATH = join(REPO_ROOT, 'scripts/ci/quarantined-tests.txt'); +const MACOS_LEDGER_PATH = join( + REPO_ROOT, + 'scripts/ci/quarantined-tests-macos.txt', +); +const WINDOWS_LEDGER_PATH = join( + REPO_ROOT, + 'scripts/ci/quarantined-tests-windows.txt', +); +const INTEGRATION_LEDGER_PATH = join( + REPO_ROOT, + 'scripts/ci/quarantined-integration-tests.txt', +); + +// Three paths quarantined by issue #2740. The merge-group flake-detection +// workflow (#1782) flagged all three with `Passed on retry` annotations in +// run 34726206593; per the recipe in +// .hermes/skills/auto-fix-issue/references/flaky-test-quarantine.md, the +// ubuntu-latest flake lands in the general ledger (no ubuntu-specific ledger +// exists; #2368 dispatch-lanes precedent) and the two macos-latest flakes +// land in the macOS ledger (windows ledger re-add policy requires a windows +// merge-group confirmed failure; sibling OS shards were green). +const ISSUE_2740_QUARANTINED_PATHS: ReadonlyArray<{ + path: string; + expectedLedger: string; +}> = [ + { + path: 'tests/unit/services/evidence-summary-adversarial.test.ts', + expectedLedger: GENERAL_LEDGER_PATH, + }, + { + path: 'tests/unit/utils/bun-compat-exit-first-2530.test.ts', + expectedLedger: MACOS_LEDGER_PATH, + }, + { + path: 'tests/unit/commands/promote-registration.test.ts', + expectedLedger: MACOS_LEDGER_PATH, + }, +]; + +// Mirror ci.yml's active-entry extraction exactly: +// grep -vE '^\s*#|^\s*$' scripts/ci/quarantined-tests-.txt +// (CRLF is normalized first so the assertion holds on any checkout config.) +function activeEntries(ledgerPath: string): string[] { + const raw = readFileSync(ledgerPath, 'utf8').replace(/\r\n/g, '\n'); + return raw + .split('\n') + .filter((line: string) => !/^\s*#/.test(line) && !/^\s*$/.test(line)) + .map((line: string) => line.trim()); +} + +describe('ci.yml integration — quarantine ledger entries for issue #2740', () => { + test.each( + ISSUE_2740_QUARANTINED_PATHS, + )('$path is an active entry in its ledger', ({ + path: quarantinedPath, + expectedLedger, + }) => { + // Regression guard for issue #2740: the merge-group flake-detection + // workflow (#1782) auto-filed all three paths as flaky candidates. + // Without these entries, Rule A would re-file duplicate issues on + // every detection (it only drops candidates already present in a + // ledger) and the unit/coverage shards would keep flaking. + expect(existsSync(expectedLedger)).toBe(true); + expect(activeEntries(expectedLedger)).toContain(quarantinedPath); + }); + + test.each( + ISSUE_2740_QUARANTINED_PATHS, + )('$path is scoped to its ledger only (no cross-ledger duplicate)', ({ + path: quarantinedPath, + expectedLedger, + }) => { + // Cross-ledger duplicates would falsely imply OS-specific or + // integration evidence: the evidence-summary flake is + // single-detection on ubuntu-latest (general ledger applies on + // every RUNNER_OS), and the two macOS flakes must NOT suppress the + // files on other OSes (the macOS ledger applies on macOS runners + // only, per the "Collect and partition test files" step). + const otherLedgers = [ + GENERAL_LEDGER_PATH, + MACOS_LEDGER_PATH, + WINDOWS_LEDGER_PATH, + INTEGRATION_LEDGER_PATH, + ].filter((ledger) => ledger !== expectedLedger); + for (const ledger of otherLedgers) { + expect(activeEntries(ledger)).not.toContain(quarantinedPath); + } + }); + + test.each( + ISSUE_2740_QUARANTINED_PATHS, + )('$path exists on disk and is discovered by the ci.yml find chain', ({ + path: quarantinedPath, + }) => { + // A typo'd ledger path would be a silent no-op: CI's comm -23 gated + // set would never exclude it (the path never appears in all-tests.txt) + // and the flake-detection workflow would keep re-filing. The unit + // discovery chain globs tests/unit/**/*.test.ts, so the on-disk file + // must exist at exactly the ledger path relative to the repo root. + expect(existsSync(join(REPO_ROOT, quarantinedPath))).toBe(true); + }); +});