fix(state-node): 同期でバージョンが失われるのを修正 - #70
Merged
Merged
Conversation
A sync stamps each operation with the timestamp of the DAG node it produced, because the receiver recomputes node CIDs from it. Both sides of that pairing were guesses based on timestamp proximity: - `get_operations` matched an operation to "the first node within ±1s", which is the genesis for every operation written in the same second. Two versions therefore shipped under one timestamp, collapsed onto a single CID at the receiver, and the newer one was lost. - `since_version` resolved to "the last operation at or before this node's timestamp", indistinguishable between versions in that same second, so an incremental catch-up could re-send or skip a version. Both are now resolved by position: `linear_history` is ordered genesis → head and the operation list is ordered by timestamp, so entry N in one is entry N in the other. Re-delivery is also no longer reported as a failure. A pull-based sync re-sends what it has, so most operations in a steady-state cluster are already held; committing one again rebuilds the same node CID, which the DAG reports as a cycle. Deployed node1 logged that 1048 times in 25 minutes while pinning a core at ~100%. Operations we already hold are now skipped by operation id. A replica left behind by the old pairing converges from an ordinary sync, so no store needs to be cleared to recover. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_012zvxAj1nZ6H1TDnkRtxnaQ
The single re-delivery case did not show the cost the deployed cluster paid: node1 logged 1048 apply failures in 25 minutes, each a full commit attempt that walks the DAG before failing, while pinning a core at ~100%. This drives 20 sync rounds from 2 providers and asserts every round accounts for all operations, so a regression shows up as a shortfall rather than as log noise nobody reads. Verified discriminating: removing the dedup check fails it (1 of 3 applied on the second round). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_012zvxAj1nZ6H1TDnkRtxnaQ
Review found two ways the previous commit fell short. **Damaged replicas did not heal.** A receiver that committed a node under a wrong CID — the actual damage the old ±1s pairing did — has the operation id recorded in state, so skipping on the id alone left the wrong node in place forever. The claim that no store needs clearing was therefore wrong for the case that matters most. Skipping now also requires the node itself to be present under the CID the stamp implies, which is recomputable from the operation. A replica damaged this way converges from an ordinary sync; `a_replica_that_committed_a_wrong_cid_ heals_from_an_ordinary_sync` fails without the check. **The positional pairing had no guard.** It holds only while the history is a line: `get_operations_with_index` returns every operation for the genesis, while `linear_history` picks one child per branch. Once the DAG forks the lists differ, and an index pairs an operation with an unrelated node — the same wrong-stamp corruption arriving through another door. A node that fails to load was also silently dropped, shifting every pair after it. Both now fail the call: a refused sync round is retried, a silently mis-stamped one is not. Also documents what `apply_operations` counts, since re-delivery makes the return value a coverage measure rather than a write count. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_012zvxAj1nZ6H1TDnkRtxnaQ
Member
Yu-da-1
approved these changes
Sep 1, 2026
somasekimoto
added a commit
that referenced
this pull request
Sep 2, 2026
The verified read let you pick any version from a dropdown, and every
choice other than the newest failed:
decrypted content does not match local_content_id ... pass the local
content id that corresponds to the version being read (HTTP 409)
The check re-derives the plaintext and compares it against the local
content id, and each version has its own. The registry keeps only the
current one, so the UI has no way to name an older version's id — the
dropdown offered choices it could not honour, and the control looked
broken rather than honest. It now reads the newest version only.
Two e2e expectations were also wrong rather than the code:
- A create writes two versions, not one (the content, then the owner's
access policy). The suite expected one because versions written in the
same second used to collapse onto a single CID — the sync bug fixed in
#70. With that fixed, two is the correct count.
- The pasted-key step read `public_key` off the gateway reply directly,
but every reply is wrapped in the SDK envelope, so the key is under
`data` and `fill()` received undefined.
Verified against the deployed cluster: J-1 and J-3 pass.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_012zvxAj1nZ6H1TDnkRtxnaQ
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
問題
同期は各 operation に「それが生成した DAG ノードの timestamp」を付けて送る（受信側はそこから CID を再計算するため）。この対応付けが timestamp の近さによる推測で、2つの不具合を起こしていた。
1. バージョンが消える —
get_operationsは「±1秒以内の最初のノード」に対応付けていた。同じ秒に書かれた operation は全て genesis にマッチするため、v2 と v3 が同一 timestamp で送られ、受信側で同じ CID に潰れて新しい方が失われる。ノード間でlatest_versionが食い違っていた原因。2. 再配信がエラー扱い — pull 型同期は持っているものを毎回送り直すので、定常状態では大半が既知の operation。再 commit すると同じ CID が再構築され DAG が cycle として拒否する。本番 node1 で 25分に1048件、CPU ほぼ100%張り付き。
since_versionの解決も同じ推測をしており、差分同期がバージョンを再送・欠落させ得た。修正
timestamp ではなく位置で対応付ける。
linear_historyは genesis→head 順、operation リストは timestamp 順なので N番目同士が対応する。since_versionも同じ履歴上の位置で解決。既知の operation は operation id（作成者が付与し operation と共に伝播する）で判定してスキップする。この位置対応が成立するのは履歴が一直線のときだけ（
get_operations_with_indexは全 operation を返すが、linear_historyは分岐点で1本しか選ばない）。DAG が分岐すると長さが食い違い、無関係なノードの timestamp を割り当ててしまう — 本PRが直したのと同じ破損が別経路で起きる。レビュー指摘を受け、長さ不一致とノード取得失敗をエラーにした。同期は再試行されるが、黙って誤った stamp を付けたものは再試行されないため。復旧について
既存ストアの削除は不要。 旧実装で壊れたレプリカは、どちらの壊れ方でも通常の同期で追いつく。
a_receiver_with_a_truncated_history_catches_upa_replica_that_committed_a_wrong_cid_heals_from_an_ordinary_sync後者はレビューで指摘を受けて対応した。当初は前者しか検証しておらず、「削除不要」の主張が過大だった。operation id を知っているだけではスキップせず、その id が示すノードが実在することも条件にしている（ノード CID は operation から再計算できる）。
検証
409テスト green、clippy 警告なし。新規テスト8本は全て、対応する修正を外すと落ちることを個別に確認済み。
ローカル4ノードの A/B も試したが、1巡目は baseline 8/9/16件・修正版0件、2巡目は両方0件とばらついた（member 選出と同期タイミング依存）。根拠はテスト側に置いている。
このPRの範囲外
本番ノードは
create中に数十秒フリーズし、ALB ヘルスチェックが落ちて ECS がタスクを入れ替える。これがゲートウェイ経由のPOST /contentが 503 になる直接の原因で、swarm loop のブロッキングという別の問題。本PRはエラーループと CPU 負荷を取り除くが、これだけでcreateが安定するわけではない。🤖 Generated with Claude Code
https://claude.ai/code/session_012zvxAj1nZ6H1TDnkRtxnaQ