Skip to content

import_replication.test: interrogate a master that will not die (#564) - #565

Merged
ohohoreilly merged 1 commit into
masterfrom
ohohoreilly/564-wedge-diagnostics
Sep 18, 2026
Merged

ohohoreilly merged 1 commit into
masterfrom
ohohoreilly/564-wedge-diagnostics

Conversation

@ohohoreilly

Copy link
Copy Markdown
Member

Groundwork for #564, not a fix for it. I could not reproduce the wedge locally in 22 runs, so this makes the next CI occurrence diagnose itself.

The problem with the evidence

GracefulShutdownUnresponsiveSlave has failed twice on the aarch64 job and neither failure could be diagnosed afterwards. Between the connection drain and TServer::Shutdown() complete, the only syslog in server.cc is the drain's own deadline warning — so a wedge in the replication stop/join, in the layer cleaners, or in the teardown jumper all produce an identical log tail. By the time anyone opens the CI output the process is long dead.

A failed Reap() is the one moment the wedged process still exists, and Reap() leaves Pid set when it times out. So interrogate it there, before the assertion that ends the fixture.

What it dumps

Per thread: state, kernel wait channel, and the raw syscall line.

wchan separates a runner loop asleep on its timer from a genuinely blocked wait. The syscall line is the more valuable half — its last two fields are the user stack pointer and PC, and in #554 the stack pointer is what proved two OS threads were executing the same stack while gdb was confidently reporting an identical bogus frame for both. On an optimised build, /proc/<pid>/task/<tid>/syscall is more trustworthy than any userspace unwinder.

Verified, not assumed

Shipping an untested diagnostic would repeat the mistake this workflow has now made twice (#559, #562). Lifted the body into a standalone probe and ran it against a live 69-thread orlyi:

tid 20090  state=S  wchan=__arm64_sys_rt_sigsuspend  syscall=133 ... 0xffffd9594e20 0xffff8391ce4c
tid 20092  state=S  wchan=futex_wait                 syscall=98  ...
tid 20093  state=S  wchan=hrtimer_nanosleep          syscall=115 ...
tid 20094  state=S  wchan=inet_csk_accept            syscall=202 ...
tid 20095  state=S  wchan=__arm64_sys_epoll_pwait    syscall=22  ...
... 69 threads reported

That is the shape that would immediately say which of the three replication loops failed to return. The fixture itself still passes 2/2 with the change in.

Best-effort and noexcept throughout — it runs on a path that has already failed and must not turn a diagnosable failure into a crash.

What this does not do

It does not fix #564. The narrowing there now points at JoinReplicationServices() blocking because a replication loop never returned, with the #461 hard-close wake chain as the suspect — but which loop, and where it is parked, is exactly what these stacks would answer and I do not have them yet.

GracefulShutdownUnresponsiveSlave has failed twice on the aarch64 job, and
neither failure could be diagnosed afterwards. The only syslog between the
connection drain and "TServer::Shutdown() complete" is the drain's own
deadline warning, so a wedge anywhere in that stretch -- the replication
stop/join, the layer cleaners, the teardown jumper -- produces an identical
log tail. By the time anyone reads the CI output the process is gone.

A failed Reap() is the one moment the wedged process still exists. Dump each
thread's state, wchan and syscall line there, before the assertion that ends
the fixture.

wchan names the kernel function a thread is blocked in, which separates a
runner loop sleeping on its timer from a genuinely blocked wait. The syscall
line matters more: its last two fields are the user stack pointer and PC, and
in #554 the stack pointer is what proved two OS threads were executing one
stack while gdb reported the same bogus frame for both. On an optimised build
that file is more trustworthy than any userspace unwinder.

Verified against a live 69-thread orlyi rather than shipped on the assumption
that it works -- it reports rt_sigsuspend for the main thread, futex_wait,
hrtimer_nanosleep for the runner loops, inet_csk_accept for the accept loop
and epoll_pwait for the discard runners. Best-effort and noexcept: this runs
on a path that has already failed and must not turn a diagnosable failure into
a crash. Fixture still passes (2/2).
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

aarch64: master wedges inside TServer::Shutdown() — GracefulShutdownUnresponsiveSlave fails intermittently on the arm job

1 participant