Repository navigation
Process-global side tables are cleared by test guards that readers are not required to take — three flakes in two days #7672
Description
Three flaky tests in two days, all the same architecture: a process-global side table, plus per-suite guards that clear it, plus tests that read it without taking the clearing lock.
| fixed in | table | how it presented |
|---|---|---|
| #7665 | opt_report's Mutex<Vec<Entry>> |
rows.len() == 2 failed at 3 — a neighbour's entry in the snapshot |
| #7665 | ext_registry's USED_PROVIDERS |
"empty after an unlisted symbol" failed with ioredis still present |
| #7671 | closure/dynamic_props.rs's CLOSURE_PROPS |
a static method value read back as TAG_UNDEFINED |
In every case the lock existed and was correct for the tests that took it. The failure is that nothing requires a reader to take it, so the defence is opt-in and the opt-in is invisible at the read site.
Each was exposed by an unrelated PR adding tests and changing the parallel schedule — never introduced by it. That is the expensive part: the author of the exposing PR spends the diagnosis, and the natural conclusion ("my change broke something") is wrong.
The remaining exposure, surveyed
Files whose tests read closure dynamic props, and whether they take crate::gc::global_side_table_test_lock():
| guarded | file (test count) |
|---|---|
| 3 of 4 | closure/dynamic_props.rs |
| 2 of 9 | object/global_this_webassembly.rs |
| 2 of 2 | object/native_module_stream.rs (after #7671) |
| 0 | array/tests.rs (65), node_stream_tests.rs (42), node_submodules/tests.rs (33), object/instanceof.rs (5), object/native_module/constants.rs (4), value/to_string.rs (4), node_stream_state_tests.rs (4), tls.rs (3), object/native_module/callable_export_arity_table.rs (3), perf_hooks.rs (2), object/prototype_chain.rs (2), object/native_call_method.rs (2), value/dynamic_object.rs (2), and 6 more with 1 each |
Roughly 180 unguarded tests. Most will never bite — the hazard needs a test that reads a persisted prop across a window a guard can land in — but "most" is doing real work in that sentence, and the three we found were each a surprise.
Options, in rough order of preference
- Make the clear observable. Have
test_clear_closure_side_tables()record that it ran, and give readers a cheap assertion that no clear happened during their read. Turns a silent wrong value into a named failure at the point of damage, without serialising anything. - Invert the default. Make the reader-side accessor take the lock, with an explicit opt-out for the guards that must clear. Costs some parallelism; removes the class.
- Scope the tables per-thread in tests. Largest change, and
CLOSURE_PROPSis keyed by heap address with a GC scanner attached, so this needs care. - Blanket-lock the ~180 tests. Rejected as the first move: it serialises a large part of the suite for a hazard whose incidence we cannot yet bound.
Option 1 is the one that pays for itself even if the class never bites again, because it converts "intermittent, diagnosed by luck" into "fails with a message naming the clearer".
Context: #7665, #7671, and gc/tests/support.rs's own comment, which documents the hazard but does not enforce anything.
Activity
docs(changelog): fragment for the guard-cleared global sinks (#7672)
proggeramlug commentedon Aug 9, 2026
Design and fix in #7674, with the survey extended: 23 tables across every family the guards clear, converted to per-thread storage in test builds via
guard_cleared_global!.Option 1 as filed is not enough on its own — a reader-side assertion is still an opt-in, and the issue's own second requirement is that a future reader must not have to remember anything. So the storage moves instead: the accessor and every call site are unchanged (
PerThread<T>derefs toT), and outside a test build the macro expands to the plainstaticbyte for byte. A lock genuinely cannot close this — the damage window is between a test's write and its read, and only the test knows that span.Option 4 stays rejected for the reason you gave. What replaces the ~180-reader burden is a
lintgate (scripts/global_sink_isolation.py) that derives the clear list fromreset_copying_nursery_runtime_test_state's own source and fails on any barestaticleft behind — the burden lands on the ~20 table authors, mechanically.Shown able to fail: reverting the macro's
#[cfg(test)]arm to the pre-fix bare static — one edit, all 23 tables — fails 7 of the 9 new tests, each naming its table. A canary declared as a plainstaticruns the same probe procedure and requires the wipe to be observed, so a green file means the detector works.Fourth instance: found, three of them, all with the split-lock-domain signature and all outside the guards' clear list, so this PR's gate is blind to them — filed as #7680. The sharpest is
async_hooks'NEXT_ASYNC_IDunder four disjoint domains withresource_ids_are_monotonic_even_without_hooksassertingb.async_id == a.async_id + 1;tui::state::SLOTSunder three, withalloc_returns_sequential_handlesassertingh0==0, h1==1, h2==2.Evidence: 25 consecutive
cargo test -p perry-runtime --lib --no-fail-fastruns green with 0 vacuous (each checked forRunning unittestsanderror[before being counted),cargo check --all-targetsclean, all 24 lint gates green.fix(test-isolation): two more per-test globals found by soaking, and …
fix(test-isolation): per-thread storage for the guard-cleared closure…
fix(test-isolation): the GC guards' clear can no longer reach another…
docs(changelog): fragment for the guard-cleared global sinks (#7672)
fix(test-isolation): two more per-test globals found by soaking, and …
fix(test-isolation): the GC guards' clear can no longer reach another…
fix(test-isolation): close the async_hooks / tui::state / timer-queue…
fix(test-isolation): close the async_hooks / tui::state / timer-queue…
docs: perry-runtime's tests must run single-threaded locally, as CI a…
docs: perry-runtime's tests must run single-threaded locally, as CI a…
docs: perry-runtime's tests must run single-threaded locally, as CI a…
docs: perry-runtime's tests must run single-threaded locally, as CI a…
lint(#10944): ratchet bare process-global statics that tests assert on
lint(#10944): ratchet bare process-global statics that tests assert on
lint(#10944): ratchet bare process-global statics that tests assert on
lint(#10944): ratchet bare process-global statics that tests assert on
test(array): make PERRY_ARRAY_INDEX_FAST_PATH_INVALIDATED per-test (#…
test(array): make PERRY_ARRAY_INDEX_FAST_PATH_INVALIDATED per-test (#…
test(gc): the control locks are test-local mutexes, not bare statics
test(gc): the control locks are test-local mutexes, not bare statics
test(gc): the control locks are test-local mutexes, not bare statics