[r3.6] docs(site): document the v3.6 plain commitment format and its migration paths - #23594
[r3.6] docs(site): document the v3.6 plain commitment format and its migration paths#23594bloxster wants to merge 3 commits into
Conversation
Snapshots Management gets a new "Commitment Snapshot Format" section: what referenced vs plain encoding means, that plain is now the default, that existing referenced files stay readable and convert during merges, and the two ways to adopt plain sooner (`erigon seg reset`, `integration commitment convert`) with their flags, destructiveness, reversibility and cost. Adds the step-size prerequisite for datadirs still recorded at the pre-3.4 step size (`erigon seg step-rebase`). CLI Reference: `--commitment.plainValues` said "Default: false", which reads as "references by default" and is wrong for v3.6. The flag is a tri-state override — omitted means plain, and only an explicit `--commitment.plainValues=false` selects the referenced encoding.
…er code fact-check Fact-checked every operator-facing claim in the v3.6 snapshot-format docs against the release/3.6 Go code. Two corrections: - seg reset: removal of a preverified .torrent with a mismatching infohash (and retraction of the data file it vouched for) is unconditional in Reset.decideRemove/doSnapshots (db/datadir/reset/reset.go), not gated by --local as the prose implied. --local=false is therefore not fully non-destructive; say so explicitly. - seg step-rebase: the exclusive chaindata open happens only in the --keep-blocks resetExecState path (cmd/utils/app/step_cmd.go); the unconditional part is the datadir flock. Also drop the space before the custom heading anchor so the generated llms-full.txt line carries no trailing whitespace, and regenerate the llms artifacts for this branch.
There was a problem hiding this comment.
Pull request overview
This PR updates Erigon’s documentation to explain the v3.6 commitment snapshot encoding change (referenced → plain by default) and to document operator migration paths (lazy merge conversion, erigon seg reset, and offline integration commitment convert), plus the prerequisite seg step-rebase path for legacy step sizes.
Changes:
- Add a new “Commitment Snapshot Format (New in v3.6)” section to Snapshots Management, including operational guidance and migration paths.
- Clarify the tri-state semantics and effective defaults of
--commitment.plainValues, including how downloader-deliverederigondb.tomlinteracts with the flag. - Update the “LLMs” index copies (
llms*.txtand docs static variants) to reflect the new scope and content.
Reviewed changes
Copilot reviewed 6 out of 6 changed files in this pull request and generated 1 comment.
Show a summary per file
| File | Description |
|---|---|
| llms.txt | Updates the Snapshots Management page summary to mention commitment + step-size migrations. |
| llms-full.txt | Adds the new commitment snapshot format section and updates the --commitment.plainValues flag description. |
| docs/site/static/llms.txt | Mirrors the Snapshots Management summary update for the docs static bundle. |
| docs/site/static/llms-full.txt | Mirrors the full-page content additions/edits for the docs static bundle. |
| docs/site/docs/fundamentals/snapshots-management.mdx | Adds the new commitment snapshot format section (core operator-facing documentation). |
| docs/site/docs/fundamentals/configuring-erigon.mdx | Updates the --commitment.plainValues flag documentation and cross-links to the new section. |
💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.
There was a problem hiding this comment.
Pull request overview
Copilot reviewed 6 out of 6 changed files in this pull request and generated 6 comments.
Suppressed comments (9)
docs/site/docs/fundamentals/snapshots-management.mdx:226
- The claim that this command is safe to rerun after an interruption is unsafe.
applyStepRebasePlanwrites the new settings only after renaming every file, whilecollectRenameListrescales every filename it currently finds. If the process stops midway, the old step size remains recorded and a rerun rescales already-renamed files a second time, potentially corrupting ranges. Remove this guarantee and direct operators to restore/recover a partially renamed datadir.
The command prints the full rename and delete plan and asks for confirmation (`Proceed with these changes? [y/N]`) before touching anything, and exits as a no-op if the datadir is already at the requested step size. It is **destructive and has no built-in undo**, but it is safe to re-run after an interruption. The node must be **stopped** — it locks the datadir (and, with `--keep-blocks`, additionally opens `chaindata/` exclusively). The v3.4.0 release notes put the rebase itself at roughly ten seconds; the cost that follows is re-acquiring whatever was in `chaindata/`.
llms-full.txt:2073
- The claim that this command is safe to rerun after an interruption is unsafe.
applyStepRebasePlanwrites the new settings only after renaming every file, whilecollectRenameListrescales every filename it currently finds. If the process stops midway, the old step size remains recorded and a rerun rescales already-renamed files a second time, potentially corrupting ranges. Remove this guarantee and direct operators to restore/recover a partially renamed datadir.
The command prints the full rename and delete plan and asks for confirmation (`Proceed with these changes? [y/N]`) before touching anything, and exits as a no-op if the datadir is already at the requested step size. It is **destructive and has no built-in undo**, but it is safe to re-run after an interruption. The node must be **stopped** — it locks the datadir (and, with `--keep-blocks`, additionally opens `chaindata/` exclusively). The v3.4.0 release notes put the rebase itself at roughly ten seconds; the cost that follows is re-acquiring whatever was in `chaindata/`.
docs/site/static/llms-full.txt:2073
- The claim that this command is safe to rerun after an interruption is unsafe.
applyStepRebasePlanwrites the new settings only after renaming every file, whilecollectRenameListrescales every filename it currently finds. If the process stops midway, the old step size remains recorded and a rerun rescales already-renamed files a second time, potentially corrupting ranges. Remove this guarantee and direct operators to restore/recover a partially renamed datadir.
The command prints the full rename and delete plan and asks for confirmation (`Proceed with these changes? [y/N]`) before touching anything, and exits as a no-op if the datadir is already at the requested step size. It is **destructive and has no built-in undo**, but it is safe to re-run after an interruption. The node must be **stopped** — it locks the datadir (and, with `--keep-blocks`, additionally opens `chaindata/` exclusively). The v3.4.0 release notes put the rebase itself at roughly ten seconds; the cost that follows is re-acquiring whatever was in `chaindata/`.
docs/site/docs/fundamentals/snapshots-management.mdx:185
- With
--nibbles.v2omitted, the converter targets V1; it does not unconditionally leave the existing key encoding unchanged. A file currently using V2 will be rewritten to V1, so this sentence understates what the example command can change.
Both encoding flags default to `false`, and `false` is the target state that matches what v3.6 writes — so the command above, with no encoding flags, converts referenced files to plain and leaves the key encoding alone.
llms-full.txt:2032
- With
--nibbles.v2omitted, the converter targets V1; it does not unconditionally leave the existing key encoding unchanged. A file currently using V2 will be rewritten to V1, so this sentence understates what the example command can change.
Both encoding flags default to `false`, and `false` is the target state that matches what v3.6 writes — so the command above, with no encoding flags, converts referenced files to plain and leaves the key encoding alone.
docs/site/static/llms-full.txt:2032
- With
--nibbles.v2omitted, the converter targets V1; it does not unconditionally leave the existing key encoding unchanged. A file currently using V2 will be rewritten to V1, so this sentence understates what the example command can change.
Both encoding flags default to `false`, and `false` is the target state that matches what v3.6 writes — so the command above, with no encoding flags, converts referenced files to plain and leaves the key encoding alone.
llms-full.txt:2049
- Conversion is driven by both target axes, not only whether values are plain. A plain file using V2 keys is also rewritten when the documented default
--nibbles.v2=falsetarget is used, so this cost estimate should refer to files not already in the full plain/V1 target state.
**What does it cost?** Every commitment `.kv` file that is not already plain is rewritten and its accessors (`.kvi` / `.bt` / `.kvei`) rebuilt. Converted output is written into `snapshots/rebuild/domain/` while the originals are still in place, so at peak you need free space for a second copy of the commitment domain — use the `commitment` row of `erigon seg du --datadir <path> -v` to size it. Plain output is larger than the referenced input.
docs/site/static/llms-full.txt:2049
- Conversion is driven by both target axes, not only whether values are plain. A plain file using V2 keys is also rewritten when the documented default
--nibbles.v2=falsetarget is used, so this cost estimate should refer to files not already in the full plain/V1 target state.
**What does it cost?** Every commitment `.kv` file that is not already plain is rewritten and its accessors (`.kvi` / `.bt` / `.kvei`) rebuilt. Converted output is written into `snapshots/rebuild/domain/` while the originals are still in place, so at peak you need free space for a second copy of the commitment domain — use the `commitment` row of `erigon seg du --datadir <path> -v` to size it. Plain output is larger than the referenced input.
docs/site/docs/fundamentals/snapshots-management.mdx:202
- Conversion is driven by both target axes, not only whether values are plain. A plain file using V2 keys is also rewritten when the documented default
--nibbles.v2=falsetarget is used, so this cost estimate should refer to files not already in the full plain/V1 target state.
**What does it cost?** Every commitment `.kv` file that is not already plain is rewritten and its accessors (`.kvi` / `.bt` / `.kvei`) rebuilt. Converted output is written into `snapshots/rebuild/domain/` while the originals are still in place, so at peak you need free space for a second copy of the commitment domain — use the `commitment` row of `erigon seg du --datadir <path> -v` to size it. Plain output is larger than the referenced input.
|
|
||
| `seg reset` is **destructive and not reversible**: at the default `--local=true` it deletes `chaindata/`, the Heimdall / Polygon-bridge DBs, and any locally generated snapshot files. Independently of `--local` — that is, even with `--local=false` — it also deletes any `.torrent` file whose infohash does not match the preverified list, **together with the snapshot data file that torrent vouched for** (a stale local build is unverifiable, so the downloader must re-fetch the canonical copy). All other preverified snapshots already on disk are kept and corrected by the downloader on the next start. The node must be **stopped** — the command takes an exclusive lock on the datadir. | ||
|
|
||
| Cost: whatever the downloader has to re-fetch, plus rebuilding state on the next start. Run `--dry-run` first, and back up the datadir. |
There was a problem hiding this comment.
This bounds the cost by the downloader, but the slow tail isn't the downloader.
Blocks freeze into torrent-restorable segments only up to keep = 1024 behind tip (block_snapshots.go:215), and the published preverified manifest can lag the live tip by much more. Anything synced past what the manifest covers exists only in chaindata/, which --local=true deletes unconditionally. That range comes back through Caplin's per-block beacon backfill, not a segment fetch — orders of magnitude slower, and on a node far enough behind it takes a second restart before execution clears ExecutionStatusTooFarAway.
Worth one line so the reader sizes the risk by how far past the manifest they are, rather than by download time.
There was a problem hiding this comment.
Fixed in 64fcab3, and the framing is now yours: size the cost by how far the node is past the published manifest, not by download time.
One correction to the citation, though — I checked block_snapshots.go before writing it. Freezing is bounded by CanRetire, which uses dbg.MaxReorgDepth (96) plus 1000-block segment alignment. The 1,024 in CanDeleteTo governs when chaindata copies of already-frozen blocks may be deleted, not when blocks freeze. So the paragraph states the manifest-coverage point without attributing it to that function.
Kept: only the manifest-covered range is torrent-restorable, --local=true removes both chaindata/ and locally built segments, and the tail returns via Caplin per-block backfill rather than a segment fetch. I softened the second-restart claim to "may need more than one restart" since I could not confirm the ExecutionStatusTooFarAway sequence from source alone — it rests on your live test.
… reset cost - Referenced encoding needs both a sub-v2.2 stamp and a >=2-step range (CommitmentBranchReferenced), so a single-step v2.1 file is plain. - --chain is a root-level flag and resolves in either position. - Size seg reset by how far the node is past the preverified manifest: segments freeze only to 1,024 behind tip, and the tail returns through Caplin per-block backfill, not a segment fetch.
Documents the v3.6 plain commitment format and the two ways to migrate an existing datadir.
Format (
/fundamentals/snapshots-management) — v3.6 writes commitment branches with keys inline instead of as offsets into the accounts and storage files. The encoding is a property of each file, not of the current setting:CommitmentBranchReferencedrequires both a version belowv2.2and a span of at leastcommitment.DefaultKeyReferencingMinStepssteps, so a small sub-v2.2file is already plain despite its stamp. Existing files stay readable and ordinary background merges migrate the datadir on their own.Migration paths —
erigon seg reset(re-fetch against the preverified set) andintegration commitment convert(offline rewrite).seg resetis destructive: at the default--local=trueit removeschaindata/, the Heimdall and Polygon-bridge DBs, and locally built snapshots. The cost is sized by how far the node is past the published preverified manifest — only the range it covers can be restored from a torrent, and the rest returns through Caplin's per-block beacon backfill.seg resethas no--chainflag of its own and reads the chain fromchaindata/;--chainis a root-level flag that resolves in either position.Config (
/fundamentals/configuring-erigon) — documents--commitment.plainValuesas the one-time, per-datadir choice recorded insnapshots/erigondb.toml, ignored once that file exists.