Skip to content

import_replication.test: interrogate the servers when an RPC deadline expires (#574) - #575

Merged
ohohoreilly merged 1 commit into
masterfrom
ohohoreilly/574-interrogate-on-rpc-deadline
Oct 2, 2026
Merged

ohohoreilly merged 1 commit into
masterfrom
ohohoreilly/574-interrogate-on-rpc-deadline

Conversation

@ohohoreilly

Copy link
Copy Markdown
Member

Makes the next occurrence of #574 diagnose itself. Test-only, no engine change.

Why

The fixture's 240s RPC deadline (#546) fired once on aarch64: the master stopped answering NewFastPrivatePov. The throw unwound into teardown, which SIGKILLed every server, so the run kept only log tails. Those can't tell a wedged master from a dead one, or either from a stuck client i/o thread like #537's. The full reasoning is in #574.

What

  • TChildServer keeps a registry of live children.
  • Before AwaitAnswered throws, it prints, for each child, Describe() (alive, state [S] or killed by signal N) and per-thread wchan/syscall. Then it prints the same for the test process's own threads.
  • import_replication.test: interrogate a master that will not die (#564) #565's DumpWedge is split so the per-thread half (DumpThreads) also reads /proc/self. It stays noexcept and best-effort.

Proven to fire

A throwaway branch forced a 0s deadline on ImportCoreVector, and ci.yml was dispatched on it (run 36958768274; branch since deleted). The make test job printed:

RPC [ImportCoreVector] unanswered after 0s (#537) -- interrogating before teardown (#574)
child 35928: alive, state [S]
  tid 35928  state=S  wchan=sigsuspend  syscall=130 0x7fff323fdd10 ...
  ... 56 more master threads ...
this test process (client side, #537):
  ... 3 threads: main, client i/o, client dispatch ...
end ImportReplication; fail

After that, the fixture failed normally and GracefulShutdownUnresponsiveSlave still ran and passed.

… expires (#574)

The fixture's 240s RPC deadline (#546) fired once on aarch64: the master
stopped answering NewFastPrivatePov. The throw then unwound into teardown,
which SIGKILLed every server, so the run kept only log tails. Those cannot
tell a wedged master from a dead one, or either from a stuck client i/o
thread like #537's.

TChildServer now keeps a registry of live children. AwaitAnswered, before
throwing, prints each child's Describe() and per-thread wchan/syscall
(#565's DumpWedge, now split so it also reads /proc/self), plus the test
process's own threads. A child that already died reports how, e.g.
"killed by signal 11", which settles the question by itself.
@ohohoreilly ohohoreilly self-assigned this Oct 2, 2026
@ohohoreilly
ohohoreilly merged commit d25cc8c into master Oct 2, 2026
10 checks passed
@ohohoreilly
ohohoreilly deleted the ohohoreilly/574-interrogate-on-rpc-deadline branch October 2, 2026 03:37
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant