Repository navigation
import_replication.test: interrogate a master that will not die (#564) - #565
Merged
Merged
Conversation
GracefulShutdownUnresponsiveSlave has failed twice on the aarch64 job, and neither failure could be diagnosed afterwards. The only syslog between the connection drain and "TServer::Shutdown() complete" is the drain's own deadline warning, so a wedge anywhere in that stretch -- the replication stop/join, the layer cleaners, the teardown jumper -- produces an identical log tail. By the time anyone reads the CI output the process is gone. A failed Reap() is the one moment the wedged process still exists. Dump each thread's state, wchan and syscall line there, before the assertion that ends the fixture. wchan names the kernel function a thread is blocked in, which separates a runner loop sleeping on its timer from a genuinely blocked wait. The syscall line matters more: its last two fields are the user stack pointer and PC, and in #554 the stack pointer is what proved two OS threads were executing one stack while gdb reported the same bogus frame for both. On an optimised build that file is more trustworthy than any userspace unwinder. Verified against a live 69-thread orlyi rather than shipped on the assumption that it works -- it reports rt_sigsuspend for the main thread, futex_wait, hrtimer_nanosleep for the runner loops, inet_csk_accept for the accept loop and epoll_pwait for the discard runners. Best-effort and noexcept: this runs on a path that has already failed and must not turn a diagnosable failure into a crash. Fixture still passes (2/2).
This was referenced Sep 18, 2026
This was referenced Oct 2, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Groundwork for #564, not a fix for it. I could not reproduce the wedge locally in 22 runs, so this makes the next CI occurrence diagnose itself.
The problem with the evidence
GracefulShutdownUnresponsiveSlavehas failed twice on the aarch64 job and neither failure could be diagnosed afterwards. Between the connection drain andTServer::Shutdown() complete, the onlysysloginserver.ccis the drain's own deadline warning — so a wedge in the replication stop/join, in the layer cleaners, or in the teardown jumper all produce an identical log tail. By the time anyone opens the CI output the process is long dead.A failed
Reap()is the one moment the wedged process still exists, andReap()leavesPidset when it times out. So interrogate it there, before the assertion that ends the fixture.What it dumps
Per thread: state, kernel wait channel, and the raw
syscallline.wchanseparates a runner loop asleep on its timer from a genuinely blocked wait. Thesyscallline is the more valuable half — its last two fields are the user stack pointer and PC, and in #554 the stack pointer is what proved two OS threads were executing the same stack while gdb was confidently reporting an identical bogus frame for both. On an optimised build,/proc/<pid>/task/<tid>/syscallis more trustworthy than any userspace unwinder.Verified, not assumed
Shipping an untested diagnostic would repeat the mistake this workflow has now made twice (#559, #562). Lifted the body into a standalone probe and ran it against a live 69-thread
orlyi:That is the shape that would immediately say which of the three replication loops failed to return. The fixture itself still passes 2/2 with the change in.
Best-effort and
noexceptthroughout — it runs on a path that has already failed and must not turn a diagnosable failure into a crash.What this does not do
It does not fix #564. The narrowing there now points at
JoinReplicationServices()blocking because a replication loop never returned, with the #461 hard-close wake chain as the suspect — but which loop, and where it is parked, is exactly what these stacks would answer and I do not have them yet.