CI flake: Compiled extensions (Windows) EPERM renaming the windows-x64 release directory

[code-yeongyu]

Summary

Job Compiled extensions (Windows), step "Verify relocated compiled extensions", test scripts/compiled-extension-load.test.ts (unnamed, ~14.6s) fails after scripts/build-binaries.sh returns 0: renameSync of the just-built release/windows-x64 directory to the relocated fixture path throws EPERM: operation not permitted.

Where

scripts/compiled-extension-load.test.ts beforeAll (~line 50):

const build = spawnSync("bash", ["scripts/build-binaries.sh", "--skip-install", "--skip-build", "--platform", platform, "--out", ...]);
expect(build.status, ...).toBe(0);
renameSync(join(release, platform), relocated);

relocated is .../relocated # % binary (Windows-legal special characters from compiledExtensionPlatform).

Reproduction

Intermittent on windows-latest. Same tree can pass or fail.

• Branch feat/omp-s2-20260913 at 249225d72 (pure merge of main at 5fb5d638d): https://github.com/code-yeongyu/senpi/actions/runs/34982822646
• main at 5fb5d638d: https://github.com/code-yeongyu/senpi/actions/runs/34955867964

Newer main runs at c7b41b1bf … 9432e4e10 passed, so this is not branch-specific.

Expected

After spawnSync reaps build-binaries.sh, the test can move the platform directory to the relocated path and then load compiled extensions from that path. A transient Windows sharing violation is retried until the handle is gone, or the test fails with the original EPERM/EBUSY/ENOTEMPTY attached if the directory never becomes movable.

Actual

error: EPERM: operation not permitted, rename '...\\release\\windows-x64' -> '...\\relocated # % binary'
      at <anonymous> (scripts/compiled-extension-load.test.ts:50:2)

build.status === 0. The unnamed test dies in beforeAll in ~15s.

Root cause hypothesis

Windows MoveFileEx returns EPERM/EBUSY while any handle still holds the directory or a just-written executable. build-binaries.sh compiles pi.exe, zips/unzips the tree, then smoke-standalone-binary.mjs actually runs that pi.exe from the directory the test immediately renames. spawnSync waits for bash, but a grandchild, an antivirus scan, or the image mapping of the just-run executable can still hold the tree for a short window. A single renameSync with no retry turns that window into a flake.

Scope and acceptance criteria

• The test still relocates the built tree away from release/<platform> and still asserts compiled-extension load from the special-character path. No .skip, no retry: N on the test, no fixed sleep, no loosened assertion.
• Rename retries only on EPERM / EBUSY / ENOTEMPTY, with short backoff and a hard deadline. Deadline expiry fails loudly with the underlying error attached (cause). Other errno codes throw immediately.
• Unit tests: injected renameSync throws EPERM twice then succeeds; injected renameSync throws EPERM forever and the helper fails with the original error attached. Clock/sleep injected so the tests are deterministic.
• Compiled extensions (Windows) is green on three consecutive runs of the same head.

Related

Same class as other Windows sharing-violation flakes: the operation is correct; the handle has not closed yet.

[code-yeongyu]

Another occurrence on the changelog-only PR #1723 (head 3f79c83, run 34985784949): Compiled extensions (Windows) step Verify relocated compiled extensions failed at 15:05:34Z while the six most recent main runs have that job green. Will rerun the failed job once the run settles.

[code-yeongyu]

Fixed by #1727, merge commit 4d441c4.

Root cause: scripts/compiled-extension-load.test.ts renamed the freshly built release directory immediately after scripts/build-binaries.sh returned; on Windows the OS still held a handle on the just-written tree for a short window, so renameSync raced it and failed with EPERM. It reproduced on main (5fb5d63) and on a pure main merge (run 34982822646) and passed on the next five main runs — a race, not a code path.

Fix: scripts/rename-sync-retry.mjs retries renameSync only on EPERM/EBUSY/ENOTEMPTY with capped backoff until a hard deadline, then fails loudly with the original errno as cause/code; the test relocates through it and also surfaces spawnSync launch errors. No assertion, timeout or scheduling changed.

Evidence: unit tests 3/3 with an injected clock (retry-twice-then-succeed; deadline expiry fails loudly with the original EPERM attached; non-transient ENOENT rethrown immediately with zero sleeps). Compiled extensions (Windows) green three consecutive times at 35cebdc (run 34997508588 attempts 1, 2, 3); full CI green.