fix: Auto-detected flaky tests (merge-group) — review for quarantine (#2740)

docs/releases/pending/issue-2740-three-merge-group-flaky-tests-quarantine.md

@@ -0,0 +1,81 @@
+# Quarantine three merge-group flaky tests (issue #2740)
+
+## What changed
+
+- Appended three new entries to the CI quarantine ledgers, each carrying the
+  structured `# OWNER:` / `# EXPIRY:` metadata block required by
+  `scripts/check-invariants.ts` Check 7 (issue #2477):
+  - `tests/unit/services/evidence-summary-adversarial.test.ts` → general
+    ledger, `scripts/ci/quarantined-tests.txt` (ubuntu-latest flake; no
+    per-OS ledger exists for ubuntu, matching the #2368 dispatch-lanes
+    precedent)
+  - `tests/unit/utils/bun-compat-exit-first-2530.test.ts` → macOS ledger,
+    `scripts/ci/quarantined-tests-macos.txt`
+  - `tests/unit/commands/promote-registration.test.ts` → macOS ledger,
+    `scripts/ci/quarantined-tests-macos.txt`
+- No source or hook code changed. The two ledger files are the only
+  repository files modified, plus a consumer-side regression test
+  (`tests/unit/scripts/ci/ci-yml-quarantine-2740.test.ts`) that pins the three
+  new ledger entries, and this pending release fragment.
+
+## Why
+
+Issue #2740 was auto-filed by the `flake-detection` workflow (issue #1782,
+`scripts/ci/detect-and-quarantine-flakes.sh`) after a merge-group CI run
+reported all three paths as flaky candidates (each with a `Passed on retry`
+annotation, so the ci.yml retry loop discarded the attempt-1 output — no
+assertion text exists to drive a root-cause fix).
+
+Triggering merge-group run 34726206593 (filed by flake-detection run
+34728138233 at 2026-09-13T00:30Z), with the flake annotation located in each
+shard's log:
+
+- `tests/unit/services/evidence-summary-adversarial.test.ts` — `unit
+  (ubuntu-latest, 2)`; attempts 1 and 2 both failed, passed on retry 2. The
+  "should not block on slow event handlers" case registers a real 100 ms
+  `setTimeout` handler and asserts
+  `expect(duration).toBeLessThan(500)` at
+  tests/unit/services/evidence-summary-adversarial.test.ts:882 — under
+  merge-group runner CPU contention the publish can exceed the 500 ms budget.
+  Sibling ubuntu/macos/windows shards green.
+- `tests/unit/utils/bun-compat-exit-first-2530.test.ts` — `unit
+  (macos-latest, 4)`; passed on retry 1. Most tests shell out to `bun build`
+  plus a `node` probe under hard 5 s timeouts
+  (`PROBE_TIMEOUT_MS` at
+  tests/unit/utils/bun-compat-exit-first-2530.test.ts:9) — a macos-latest
+  runner stall >5 s trips the probe timeout. Sibling shards green.
+- `tests/unit/commands/promote-registration.test.ts` — `unit
+  (macos-latest, 6)`; passed on retry 1. `beforeEach` mutates
+  `process.env.HOME` / `LOCALAPPDATA` / `XDG_DATA_HOME` to redirect the hive
+  knowledge path into a fresh `mkdtempSync` temp dir (cleaned in `afterEach`)
+  — the classic env-mutation + temp-dir teardown surface (#1782 class).
+  Sibling shards green.
+
+All three files pass locally on this checkout (62 pass / 0 fail across the
+three files in 4.51 s), confirming the flakes are environment-sensitive
+(merge-group runner pressure) rather than logic bugs. The `unit
+(windows-latest, 4)` failure in the same run was `close-active-state-unlink-
+retry.test.ts` — unrelated (already root-fixed surface, different file, and
+the detect job did not flag it).
+
+## Migration steps
+
+None. Quarantine is a CI-gating data change: the ci.yml unit-shard discovery
+pipeline (`grep | sort | comm`) now excludes these three paths from the gated
+test set on the relevant OS (general ledger applies to all OSes, macOS ledger
+to macos-latest only).
+
+## Known caveats
+
+- Entries carry EXPIRY 2026-10-14 (30 days out; inside the 14-day grace
+  window Check 7 only warns). They must be root-fixed or renewed before the
+  expiry closes; the recommended root fixes are noted in each entry's
+  `# EXPIRY` criterion line.
+- Ledger placement is per-OS evidence-based: the evidence-summary flake was
+  single-detection on ubuntu-latest so it landed in the general ledger (no
+  ubuntu-specific ledger exists); the two macOS flakes landed in the macOS
+  ledger. If any file later flakes on a second OS, it should be split or
+  moved accordingly.
+- These are the first macOS-ledger entries added alongside the #2738 signal
+  test; both ledgers are independent of the windows ledger (STATUS: 2
+  entries) and the general ledger (now 1 active entry).

scripts/ci/quarantined-tests-macos.txt

@@ -21,3 +21,44 @@
 # EXPIRY: 2026-09-26 — remove when the fixture/harness signal handling is
 #   fixed and the test passes on macos-latest.
 tests/unit/scripts/ci/repository-validation-real-process-2675.test.ts
+#
+# tests/unit/utils/bun-compat-exit-first-2530.test.ts (issue #2740)
+# ---------------------------------------------------------------------------
+# Regression coverage for #2530: verifies the bunSpawn Node fallback consumes
+# stdout/stderr exit-first (empty/small/large/nonzero dual-stream matrix,
+# bounded-output overflow, timeout termination) across both native Bun and the
+# plain-Node fallback, plus the public isCleanWorktree path under node
+# --input-type=module. Every non-matrix test shell outs to real processes:
+# `bun build` of src/utils/bun-compat.ts and a `node` probe, both under hard
+# 5s timeouts (PROBE_TIMEOUT_MS at tests/unit/utils/bun-compat-exit-first-2530.test.ts:9).
+# A macos-latest merge-group runner stalled by >5s on cold build/install or
+# CPU contention trips the probe timeout, which surfaces as attempt-1 failure
+# with no product defect involved. Pre-existing, unrelated to any specific PR.
+# Quarantined per issue #2740; tracked under #1782 (test-stability sprint).
+# OWNER: zaxbysauce — issue #2740 (macos-latest merge-group retry-flake,
+#   run 34726206593 `unit (macos-latest, 4)`; siblings green)
+# EXPIRY: 2026-10-14 — root-fix the probe timeout sensitivity (e.g. raise
+#   PROBE_TIMEOUT_MS or pre-warm the build) or renew with an updated
+#   criterion. Inside the 14-day grace window Check 7 only warns.
+tests/unit/utils/bun-compat-exit-first-2530.test.ts
+#
+# tests/unit/commands/promote-registration.test.ts (issue #2740)
+# ---------------------------------------------------------------------------
+# Unit tests for the `/swarm promote` command registration and behavior
+# (command import/export, switch-case registration, help text, direct-text
+# mode, --category/--from-swarm/--force/--reason parsing, and the #1821 A3
+# actionability-floor gate). beforeEach mutates process.env.HOME /
+# LOCALAPPDATA / XDG_DATA_HOME to redirect the hive knowledge path into a
+# fresh mkdtempSync temp dir, and afterEach restores it and rmSync's the dir —
+# the classic env-mutation + temp-dir teardown surface (#1782 class) plus a
+# real filesystem write to the hive path per test. Attempt-1 failure on
+# macos-latest merge-group runner with immediate retry pass is consistent
+# with cold-file / env-read sensitivity, not a product regression.
+# Pre-existing, unrelated to any specific PR.
+# Quarantined per issue #2740; tracked under #1782 (test-stability sprint).
+# OWNER: zaxbysauce — issue #2740 (macos-latest merge-group retry-flake,
+#   run 34726206593 `unit (macos-latest, 6)`; siblings green)
+# EXPIRY: 2026-10-14 — root-fix the per-test env mutation / temp-dir teardown
+#   or renew with an updated criterion. Inside the 14-day grace window
+#   Check 7 only warns.
+tests/unit/commands/promote-registration.test.ts

scripts/ci/quarantined-tests.txt

@@ -12,3 +12,27 @@
 # the grace window it only warns).
 #
 # No active global test quarantines. Issue #1908 retired the tracked backlog.
+#
+# tests/unit/services/evidence-summary-adversarial.test.ts (issue #2740)
+# ---------------------------------------------------------------------------
+# Adversarial security tests for the Evidence Summary Pipeline (Task 5.8):
+# malformed evidence payloads, blocker spoofing, artifact path abuse, and
+# event spam — exercised through buildEvidenceSummary /
+# createEvidenceSummaryIntegration with mocked loadEvidence / loadPlanJsonOnly.
+# Core timing hazard: the "should not block on slow event handlers" case
+# registers a real 100ms setTimeout handler and asserts
+# expect(duration).toBeLessThan(500) at
+# tests/unit/services/evidence-summary-adversarial.test.ts:882 — under
+# merge-group runner CPU contention (6 unit shards co-scheduled with coverage
+# and smoke jobs) the publish can exceed the 500ms budget and fail attempt 1,
+# then pass on retry when the runner has settled. The temp dir per test is
+# also mkdtempSync/rmSync'd. Pre-existing, unrelated to any specific PR.
+# Quarantined per issue #2740; tracked under #1782 (test-stability sprint).
+# OWNER: zaxbysauce — issue #2740 (ubuntu-latest merge-group retry-flake,
+#   run 34726206593 `unit (ubuntu-latest, 2)`; attempts 1+2 failed, passed on
+#   retry 2; no per-OS ledger exists for ubuntu so this uses the general one,
+#   matching the #2368 dispatch-lanes precedent)
+# EXPIRY: 2026-10-14 — root-fix the 500ms wall-clock budget (e.g. freeze the
+#   clock or widen the margin) or renew with an updated criterion. Inside the
+#   14-day grace window Check 7 only warns.
+tests/unit/services/evidence-summary-adversarial.test.ts

tests/unit/scripts/ci/ci-yml-quarantine-2740.test.ts

@@ -0,0 +1,109 @@
+import { describe, expect, test } from 'bun:test';
+import { existsSync, readFileSync } from 'node:fs';
+import { join } from 'node:path';
+
+// Repo root is four levels up from tests/unit/scripts/ci/.
+const REPO_ROOT = join(import.meta.dir, '../../../..');
+const GENERAL_LEDGER_PATH = join(REPO_ROOT, 'scripts/ci/quarantined-tests.txt');
+const MACOS_LEDGER_PATH = join(
+	REPO_ROOT,
+	'scripts/ci/quarantined-tests-macos.txt',
+);
+const WINDOWS_LEDGER_PATH = join(
+	REPO_ROOT,
+	'scripts/ci/quarantined-tests-windows.txt',
+);
+const INTEGRATION_LEDGER_PATH = join(
+	REPO_ROOT,
+	'scripts/ci/quarantined-integration-tests.txt',
+);
+
+// Three paths quarantined by issue #2740. The merge-group flake-detection
+// workflow (#1782) flagged all three with `Passed on retry` annotations in
+// run 34726206593; per the recipe in
+// .hermes/skills/auto-fix-issue/references/flaky-test-quarantine.md, the
+// ubuntu-latest flake lands in the general ledger (no ubuntu-specific ledger
+// exists; #2368 dispatch-lanes precedent) and the two macos-latest flakes
+// land in the macOS ledger (windows ledger re-add policy requires a windows
+// merge-group confirmed failure; sibling OS shards were green).
+const ISSUE_2740_QUARANTINED_PATHS: ReadonlyArray<{
+	path: string;
+	expectedLedger: string;
+}> = [
+	{
+		path: 'tests/unit/services/evidence-summary-adversarial.test.ts',
+		expectedLedger: GENERAL_LEDGER_PATH,
+	},
+	{
+		path: 'tests/unit/utils/bun-compat-exit-first-2530.test.ts',
+		expectedLedger: MACOS_LEDGER_PATH,
+	},
+	{
+		path: 'tests/unit/commands/promote-registration.test.ts',
+		expectedLedger: MACOS_LEDGER_PATH,
+	},
+];
+
+// Mirror ci.yml's active-entry extraction exactly:
+//   grep -vE '^\s*#|^\s*$' scripts/ci/quarantined-tests-<os>.txt
+// (CRLF is normalized first so the assertion holds on any checkout config.)
+function activeEntries(ledgerPath: string): string[] {
+	const raw = readFileSync(ledgerPath, 'utf8').replace(/\r\n/g, '\n');
+	return raw
+		.split('\n')
+		.filter((line: string) => !/^\s*#/.test(line) && !/^\s*$/.test(line))
+		.map((line: string) => line.trim());
+}
+
+describe('ci.yml integration — quarantine ledger entries for issue #2740', () => {
+	test.each(
+		ISSUE_2740_QUARANTINED_PATHS,
+	)('$path is an active entry in its ledger', ({
+		path: quarantinedPath,
+		expectedLedger,
+	}) => {
+		// Regression guard for issue #2740: the merge-group flake-detection
+		// workflow (#1782) auto-filed all three paths as flaky candidates.
+		// Without these entries, Rule A would re-file duplicate issues on
+		// every detection (it only drops candidates already present in a
+		// ledger) and the unit/coverage shards would keep flaking.
+		expect(existsSync(expectedLedger)).toBe(true);
+		expect(activeEntries(expectedLedger)).toContain(quarantinedPath);
+	});
+
+	test.each(
+		ISSUE_2740_QUARANTINED_PATHS,
+	)('$path is scoped to its ledger only (no cross-ledger duplicate)', ({
+		path: quarantinedPath,
+		expectedLedger,
+	}) => {
+		// Cross-ledger duplicates would falsely imply OS-specific or
+		// integration evidence: the evidence-summary flake is
+		// single-detection on ubuntu-latest (general ledger applies on
+		// every RUNNER_OS), and the two macOS flakes must NOT suppress the
+		// files on other OSes (the macOS ledger applies on macOS runners
+		// only, per the "Collect and partition test files" step).
+		const otherLedgers = [
+			GENERAL_LEDGER_PATH,
+			MACOS_LEDGER_PATH,
+			WINDOWS_LEDGER_PATH,
+			INTEGRATION_LEDGER_PATH,
+		].filter((ledger) => ledger !== expectedLedger);
+		for (const ledger of otherLedgers) {
+			expect(activeEntries(ledger)).not.toContain(quarantinedPath);
+		}
+	});
+
+	test.each(
+		ISSUE_2740_QUARANTINED_PATHS,
+	)('$path exists on disk and is discovered by the ci.yml find chain', ({
+		path: quarantinedPath,
+	}) => {
+		// A typo'd ledger path would be a silent no-op: CI's comm -23 gated
+		// set would never exclude it (the path never appears in all-tests.txt)
+		// and the flake-detection workflow would keep re-filing. The unit
+		// discovery chain globs tests/unit/**/*.test.ts, so the on-disk file
+		// must exist at exactly the ledger path relative to the repo root.
+		expect(existsSync(join(REPO_ROOT, quarantinedPath))).toBe(true);
+	});
+});