orlyi freezes silently when the system log daemon stops reading /dev/log #641

orlyi freezes completely, and silently, if the system log daemon stops reading /dev/log.

Mechanism.

• Every log line goes through glibc's syslog().
• glibc holds one process-wide lock across both the stderr echo (LOG_PERROR, which --le turns on) and a blocking sendto() on the /dev/log socket.
• If journald (or rsyslog) stalls, the socket's queue fills. The thread inside syslog() prints its line to stderr, then blocks in sendto.
• Every other thread that logs then blocks on the lock without printing. The server stops serving and stops logging at the same moment, so the log looks like a clean hang with no cause.

Reproduced on an ubuntu-24.04-arm runner (probe run 37392296481):

• Freezing journald (SIGSTOP) during import_replication.test's GracefulShutdownUnresponsiveSlave failed it 2 times out of 2, with exactly aarch64: master wedges inside TServer::Shutdown() — GracefulShutdownUnresponsiveSlave fails intermittently on the arm job #564's signature. The control passed.
• Thread dump: one master thread in unix_wait_for_peer (sendto with MSG_NOSIGNAL), and 33 threads in futex_do_wait.
• This is the likely cause of aarch64: master wedges inside TServer::Shutdown() — GracefulShutdownUnresponsiveSlave fails intermittently on the arm job #564's one natural occurrence, and it fits import_replication.test: master stops answering NewFastPrivatePov after an expired-pov discard during a replication stall (aarch64, once) #574's evidence too.

Why it matters outside tests. A stalled log daemon is uncommon but real: journald under memory pressure, a full disk, or a slow log forwarder. Today that turns into a full server outage with no log line saying why. Inside containers it rarely applies, because the image has no /dev/log listener and syslog() then just drops the send. That's worth checking.

Options (a design decision, so not done yet):

1. A logging backend that never blocks: send to /dev/log with MSG_DONTWAIT, drop the line when the socket is full, count drops, and report the count on the reporting port. stderr keeps every line.
2. A stderr-only mode (no /dev/log) as a flag, used by the test fixtures and the container image; keep syslog() as it is otherwise.
3. Both: 2 now, 1 later.

How to recognise it next time: a thread in unix_wait_for_peer / sendto in a hang dump (the #565/#575 dumps capture wchan and the syscall line).