Fix ABBA deadlock between allocation cleanup and ICE-ufrag registration

[srperens]

Why

On 2026-05-27 the production server hung silently for ~1h40m (no crash, no panic, no log output) until a manual docker restart. Root cause: an ABBA lock-order deadlock in AllocationTable.

AllocationTable keeps a primary allocations DashMap plus secondary index maps. The cleanup_* paths hold an allocations shard write lock (via retain) and then mutate the secondary indices — order allocations → secondary. register_ice_ufrags did the reverse: it held a by_ice_ufrag write lock (entry API) and then acquired an allocations read lock via get().

On the multi-thread tokio runtime, a concurrent cleanup and register on colliding shards park both threads forever. Because packets are processed inline on the recv loop, the whole server then goes silent. It ran for ~5.5 weeks before the rare shard collision finally hit; the last log line before the freeze was Removing inactive allocation — emitted from inside cleanup_inactive's retain closure, exactly where the cleanup side parks.

What

Commit 1 — fix(table)

• Reorder register_ice_ufrags to acquire allocations first, matching the cleanup paths. The atomic by_ice_ufrag dedup and exact return semantics are preserved.
• Document the lock-order invariant on the AllocationTable struct (allocations before any secondary index, never the reverse).
• Add a watchdog'd contention regression test that drives register_ice_ufrags and cleanup_inactive from multiple threads. Validated: it deadlocks (and fails via the watchdog) on the pre-fix order, and passes after.

Commit 2 — refactor(handler)

• The handlers bound a DashMap guard and then awaited socket.send_to(...) while it was alive (handle_send held both the sender's and each target's guard across every relay send). A guard is a shard RwLock guard; holding it across network I/O blocks every writer on that shard — including the cleanup task — for the full send duration.
• Reworked handle_binding_request, handle_client_response, handle_refresh, handle_create_permission, handle_channel_bind, and handle_send to snapshot the owned data they need, drop the guard, then await. Behavior is preserved; the rare allocation-reaped-mid-request races resolve to the same error/skip paths.

Verification

• cargo clippy --all-targets clean.
• cargo test — 36 passed (35 existing + new regression test).
• Regression test validated against the buggy lock order (deadlocks → watchdog fails it).

Caveat: there are no integration tests for the WebRTC routing in handle_send, so commit 2 is verified by compile + clippy + unit tests + manual reasoning only. A WHEP smoke test after deploy is the real confirmation.

🤖 Generated with Claude Code