[ohohoreilly]

Closes #554. Unblocks #552.

What was actually wrong

Not a shutdown-ordering bug. Two OS threads were executing the same stack.

A fiber's C stack outlives the OS thread that last ran it: TFrames come from a cross-thread pool (TThreadLocalPool::Free is explicitly a cross-thread free list, and TryAllocUncommon steals from peer pools) and are routinely re-latched onto a runner belonging to a different thread. TJumpRunnable does exactly this by design — it allocates a frame from the calling thread's pool and latches it onto someone else's runner.

The compiler assumes the opposite. GCC computes the thread pointer once per function and derives every __thread access in that function from it. On aarch64 that is a single mrs x20, tpidr_el0 hoisted out of the loop:

545edc:  mrs  x20, tpidr_el0     <- computed ONCE, at function entry
545ee8:  add  x20, x20, #0x38    <- x20 = &LocalRunner in THIS thread's TLS
545ef8:  ...                     <- loop top (TFrame::Run's for(;;))
545f24:  blr  x1                 <- run the runnable (may switch threads)
545f28:  ldr  x0, [x20]          <- LocalRunner, from the CACHED address
545f30:  bl   switch_to_fiber
545f34:  b    545ef8

TFrame::Run() reads TRunner::LocalRunner after the runnable returns, to switch back to "my" scheduler. On a frame that had migrated, that read returned the previous thread's runner, and the fiber longjmped onto that thread's scheduler stack.

x86-64 folds the TLS base into each access's addressing mode, so there is nothing to hoist and nothing to go stale. That is the whole of the arm-vs-x86 difference — not scheduling luck.

How that produced the hang

The durable manager's writer host job hijacked BGFastRunner's stack, so WriterScheduler.Run() never returned and the job never reached its SchedulerExitedSem.Push(). ~TDurableManager blocked on the matching Pop() forever, and because that destructor runs on the teardown fiber, TServer::Shutdown() wedged with it.

Instrumenting the destructor showed it exactly — the merger half completes, the writer half never does:

#554 dtor writer_hosted=1 merger_hosted=1
#554 dtor WriterFinishedSem done
#554 dtor MergerFinishedSem done
#554 dtor both runners ShutDown()
#554 MERGER host RUN RETURNED
#554 MERGER host PUSHED
#554 dtor SchedulerExitedSem pop 1 done      <- and then nothing

Both runners' KeepRunning read false in the wedged process and both frame queues were empty, so the writer's loop had nothing left to do and every reason to exit — it just was not running its own loop any more.

The kernel confirms the stack sharing. Two distinct tasks, identical user stack pointer:

tid 17746 syscall: 115 0x0 0x0 0xffff9c9ce520 0xffff9c9ce520 0x0 0x0 0xffff9c9ce490 ...
tid 17754 syscall: 115 0x0 0x0 0xffff9c9ce520 0xffff9c9ce520 0x0 0x0 0xffff9c9ce490 ...

0xffff9c9ce... is tid 17746's own thread stack. 17754 is the writer host, running on it.

The fix

The three fiber thread-locals — TRunner::LocalRunner, TFrame::LocalFrame, TFrame::LocalFramePool — become TFiberSafeLocal slots whose accessors are noipa. GCC may not inline, clone, or infer anything about a noipa function, so the thread pointer is re-read at every access, and it survives -flto (verified in the built binary: StartFrame now has no mrs and calls the accessor each iteration).

Every existing call site is unchanged. The slot converts, dereferences, tests and assigns exactly as the bare pointer did; one site that passed it to a deduced template parameter (Base::AssertTrue) now spells the read out as .Get().

Nothing cheaper suppresses the hoist. Measured, not assumed — volatile, thread_local, an asm volatile memory clobber, returns_twice on switch_to_fiber, and all four -ftls-model settings each still emit one loop-invariant mrs. The thread pointer is not memory as far as the optimiser is concerned; it is a function invariant, and only an opaque call re-reads it.

Verification

On a native aarch64 container configured to match the CI runner's four-core topology (which decides the fast/slow runner counts):

	result
before, release	wedged on the first attempt, 3 out of 3 times
after, release	20 / 20 runs exit 0
after, jhm -c debug whole tree	1753 jobs, clean

Worth recording for whoever writes the regression guard: a debug build does not reproduce this — 15 pre-fix debug runs were all clean. So adding lang_test.py to the aarch64 job as #554 suggested would not by itself have caught it; the cheap guard that would is compiling one package with a release orlyc on arm, which is what the repro workflow on the #548 branch already does in about nine minutes. Filing that separately rather than bundling it here.

Scope

Deliberately limited to the fiber layer's own thread-locals, which is where the bug is and where the load-bearing decision (which stack does this thread run on) is made. The same hazard exists in principle for other thread-locals a fiber can read after a switch — TDiskController::TEvent::LocalEventPool, TLocalReadFileCache::Cache, TLocalWalkerCache::Cache. None is implicated in this hang and each is an audit of its own, so they are not touched here.