Skip to content

Make the remaining fiber-visible thread-locals fiber-safe (#578) - #579

Merged
ohohoreilly merged 1 commit into
masterfrom
ohohoreilly/578-fiber-safe-caches
Oct 2, 2026
Merged

ohohoreilly merged 1 commit into
masterfrom
ohohoreilly/578-fiber-safe-caches

Conversation

@ohohoreilly

Copy link
Copy Markdown
Member

Fixes #578.

Cause

The kernel's fault report for the v0.1.0 image puts the PC at orlyi+0x613ff4: TLocalReadFileCache<...>::Get, faulting on its first load from this, with this == nullptr. Its caller is the lambda in TSafeRepo::RemoveFile that visits each scheduler to clear its read-file cache:

Fiber::SwitchTo(runner);
... TLocalReadFileCache<...>::Cache->Get(...);

In the shipped binary that lambda does mrs x27, tpidr_el0 at entry (0x6144c8), calls switch_to_fiber (0x614524), and then derives Cache's address from the stale x27 (0x61452c). After the switch the fiber is running on the target scheduler's thread, but it reads the calling thread's slot, which is null there. This is the #554 hazard on one of the three thread-locals that #555 deliberately left unaudited. amd64 is unaffected because x86-64 folds the TLS base into each access.

Fix

The three remaining fiber-visible thread-locals become TFiberSafeLocal, the same as TRunner::LocalRunner and friends:

  • TLocalReadFileCache::Cache
  • TLocalWalkerCache::Cache (read in the same lambda)
  • TDiskController::TEvent::LocalEventPool

TJumpRunnable::EnsureLocalDiskEventPool bound a pool_t *& to the slot. A reference caches the thread pointer the same way, so it now goes through the accessor. Call sites are otherwise unchanged.

TReadFile::HashHitCount is still a bare __thread. It's a statistics counter, so a wrong-thread increment there miscounts but can't crash.

Verification

Native aarch64 (Docker on Apple Silicon), release build with LTO:

  • Disassembly: the fixed RemoveFile lambda has no mrs, and it calls TFiberSafeLocal<...>::Get() after switch_to_fiber. Current master still hoists the mrs.
  • Behaviour: each run starts the image's orlyi command line, writes sample.write_val through a safe shared, a fast shared and a safe private POV, then checks the server is still alive.
build runs died (139)
master, image packages 10 10
master, freshly compiled packages 5 5
this branch 20 0

CI's aarch64 job builds debug, which doesn't hoist, so it can't catch this class. That gap is #556.

The v0.1.0 image segfaulted on linux/arm64 right after the first write
through a POV. The faulting PC is TLocalReadFileCache::Get with a null
this, called from the TSafeRepo::RemoveFile lambda that visits every
scheduler to clear its read-file cache. That lambda reads tpidr_el0 once
at entry, SwitchTo()s onto the target scheduler's thread, then derives
Cache's address from the stale thread pointer, so it reads the calling
thread's slot, which is null there. It is the #554 hazard, on one of the
three thread-locals #555 left unaudited.

Convert all three to TFiberSafeLocal: TLocalReadFileCache::Cache,
TLocalWalkerCache::Cache and TDiskController::TEvent::LocalEventPool.
EnsureLocalDiskEventPool bound a reference to the slot, which caches the
thread pointer just the same, so it now goes through the accessor.

Verified on native aarch64 (release, LTO): the lambda no longer has an
mrs, and calls the accessor after switch_to_fiber. A loop that writes
through safe shared, fast shared and safe private POVs killed the
unfixed build 10/10 times (5/5 with freshly compiled packages) and the
fixed build 0/20.
@ohohoreilly ohohoreilly self-assigned this Oct 2, 2026
@ohohoreilly
ohohoreilly merged commit e78c414 into master Oct 2, 2026
10 checks passed
@ohohoreilly
ohohoreilly deleted the ohohoreilly/578-fiber-safe-caches branch October 2, 2026 11:31
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

v0.1.0 image segfaults on the first write on linux/arm64 (release build; amd64 unaffected)

1 participant