fix(state-node): 容量問い合わせを並行化する - #71
Merged
Merged
Conversation
`create` asks every placement candidate for its capacity before choosing members. The queries were sent in sequence, so each unreachable peer cost a full 30s timeout before the next was even asked — the caller waited timeout × N. Deployed node1 shows exactly that shape: 30.4s of silence, then 61.0s (two timeouts), each followed by "Outbound request failed: Timeout" and "member(s) did not answer the capacity query". A request stalled that long outlives the ALB health check, so ECS replaces the task mid-request. Both batch helpers now dispatch their queries together and await them together, so a batch costs one timeout regardless of how many peers stay silent. `query_node_public_keys_batch` had the same defect and is fixed with it. Capacity queries also get their own, much shorter timeout (5s). Silence there is cheap to act on — the peer is left out of the candidate set and another is used — so holding a client request for the full network timeout to hear it buys nothing. Tests use a stubbed swarm loop that accepts queries and never answers. Verified discriminating: with the batch restored to sequential (keeping the shorter timeout, so only the serialisation differs) three silent peers take 15s and the tests fail. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_012zvxAj1nZ6H1TDnkRtxnaQ
Yu-da-1
approved these changes
Sep 1, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
問題
createは配置先を決める前に、候補 peer 全員へ容量を問い合わせる。この問い合わせが逐次で、応答しない peer ごとに 30秒フルに待っていた。呼び出し側はtimeout × N待つことになる。本番 node1 のログはこの形そのもの。
空白の直後には必ず
Outbound request failed: Timeoutとmember(s) did not answer the capacity queryが続く。これだけ止まると ALB のヘルスチェックが落ち、ECS がリクエスト処理中のタスクを入れ替える。修正
2つの batch helper が、問い合わせをまとめて送信しまとめて待つようにした。無応答 peer が何台いても、batch 全体で1回分のタイムアウトに収まる。
query_node_public_keys_batchも同じ欠陥だったので併せて修正。容量問い合わせには専用の短いタイムアウト（5秒）を設けた。ここでの無応答は対処が軽く（候補から外して別の peer を使うだけ）、それを聞くためにクライアントのリクエストを 30秒保留する意味がないため。
検証
404テスト green、clippy 警告なし。テストは swarm loop をスタブ化し、問い合わせを受け付けるが決して応答しない peer を再現する。
差別性を確認済み: batch を逐次に戻すと（短いタイムアウトは残したまま＝並行化の有無だけが違う状態）、無応答3台で 15秒かかりテストが落ちる。
なおローカル実機での所要時間比較も試みたが、gateway 側のタイムアウトや state-node のレート制限が先に効いてしまい、state-node 側の差を切り分けられなかった。根拠はテストに置いている。
関連
同じ本番障害を追う中で見つかった CRDT 同期のバグは #70 で別途対応。こちらはリクエストの停止（503 の直接原因）、#70 はバージョン欠落とエラーループで、領域が重ならないため独立してマージできる。
🤖 Generated with Claude Code
https://claude.ai/code/session_012zvxAj1nZ6H1TDnkRtxnaQ