Skip to content

[r3.6] docs(site): document the v3.6 plain commitment format and its migration paths - #23594

Open
bloxster wants to merge 3 commits into
release/3.6from
docs/v36-snapshots
Open

[r3.6] docs(site): document the v3.6 plain commitment format and its migration paths#23594
bloxster wants to merge 3 commits into
release/3.6from
docs/v36-snapshots

Conversation

@bloxster

@bloxster bloxster commented Aug 26, 2026

Copy link
Copy Markdown
Collaborator

Documents the v3.6 plain commitment format and the two ways to migrate an existing datadir.

Format (/fundamentals/snapshots-management) — v3.6 writes commitment branches with keys inline instead of as offsets into the accounts and storage files. The encoding is a property of each file, not of the current setting: CommitmentBranchReferenced requires both a version below v2.2 and a span of at least commitment.DefaultKeyReferencingMinSteps steps, so a small sub-v2.2 file is already plain despite its stamp. Existing files stay readable and ordinary background merges migrate the datadir on their own.

Migration pathserigon seg reset (re-fetch against the preverified set) and integration commitment convert (offline rewrite). seg reset is destructive: at the default --local=true it removes chaindata/, the Heimdall and Polygon-bridge DBs, and locally built snapshots. The cost is sized by how far the node is past the published preverified manifest — only the range it covers can be restored from a torrent, and the rest returns through Caplin's per-block beacon backfill. seg reset has no --chain flag of its own and reads the chain from chaindata/; --chain is a root-level flag that resolves in either position.

Config (/fundamentals/configuring-erigon) — documents --commitment.plainValues as the one-time, per-datadir choice recorded in snapshots/erigondb.toml, ignored once that file exists.

Bloxster added 2 commits August 26, 2026 12:22
Snapshots Management gets a new "Commitment Snapshot Format" section:
what referenced vs plain encoding means, that plain is now the default,
that existing referenced files stay readable and convert during merges,
and the two ways to adopt plain sooner (`erigon seg reset`,
`integration commitment convert`) with their flags, destructiveness,
reversibility and cost. Adds the step-size prerequisite for datadirs
still recorded at the pre-3.4 step size (`erigon seg step-rebase`).

CLI Reference: `--commitment.plainValues` said "Default: false", which
reads as "references by default" and is wrong for v3.6. The flag is a
tri-state override — omitted means plain, and only an explicit
`--commitment.plainValues=false` selects the referenced encoding.
…er code fact-check

Fact-checked every operator-facing claim in the v3.6 snapshot-format docs
against the release/3.6 Go code. Two corrections:

- seg reset: removal of a preverified .torrent with a mismatching infohash
  (and retraction of the data file it vouched for) is unconditional in
  Reset.decideRemove/doSnapshots (db/datadir/reset/reset.go), not gated by
  --local as the prose implied. --local=false is therefore not fully
  non-destructive; say so explicitly.

- seg step-rebase: the exclusive chaindata open happens only in the
  --keep-blocks resetExecState path (cmd/utils/app/step_cmd.go); the
  unconditional part is the datadir flock.

Also drop the space before the custom heading anchor so the generated
llms-full.txt line carries no trailing whitespace, and regenerate the
llms artifacts for this branch.

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

This PR updates Erigon’s documentation to explain the v3.6 commitment snapshot encoding change (referenced → plain by default) and to document operator migration paths (lazy merge conversion, erigon seg reset, and offline integration commitment convert), plus the prerequisite seg step-rebase path for legacy step sizes.

Changes:

  • Add a new “Commitment Snapshot Format (New in v3.6)” section to Snapshots Management, including operational guidance and migration paths.
  • Clarify the tri-state semantics and effective defaults of --commitment.plainValues, including how downloader-delivered erigondb.toml interacts with the flag.
  • Update the “LLMs” index copies (llms*.txt and docs static variants) to reflect the new scope and content.

Reviewed changes

Copilot reviewed 6 out of 6 changed files in this pull request and generated 1 comment.

Show a summary per file
File Description
llms.txt Updates the Snapshots Management page summary to mention commitment + step-size migrations.
llms-full.txt Adds the new commitment snapshot format section and updates the --commitment.plainValues flag description.
docs/site/static/llms.txt Mirrors the Snapshots Management summary update for the docs static bundle.
docs/site/static/llms-full.txt Mirrors the full-page content additions/edits for the docs static bundle.
docs/site/docs/fundamentals/snapshots-management.mdx Adds the new commitment snapshot format section (core operator-facing documentation).
docs/site/docs/fundamentals/configuring-erigon.mdx Updates the --commitment.plainValues flag documentation and cross-links to the new section.

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

Comment thread docs/site/docs/fundamentals/snapshots-management.mdx
@bloxster
bloxster marked this pull request as ready for review August 26, 2026 11:50
@yperbasis
yperbasis requested review from awskii and a balanced review from Copilot August 26, 2026 13:43

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Copilot reviewed 6 out of 6 changed files in this pull request and generated 6 comments.

Suppressed comments (9)

docs/site/docs/fundamentals/snapshots-management.mdx:226

  • The claim that this command is safe to rerun after an interruption is unsafe. applyStepRebasePlan writes the new settings only after renaming every file, while collectRenameList rescales every filename it currently finds. If the process stops midway, the old step size remains recorded and a rerun rescales already-renamed files a second time, potentially corrupting ranges. Remove this guarantee and direct operators to restore/recover a partially renamed datadir.
The command prints the full rename and delete plan and asks for confirmation (`Proceed with these changes? [y/N]`) before touching anything, and exits as a no-op if the datadir is already at the requested step size. It is **destructive and has no built-in undo**, but it is safe to re-run after an interruption. The node must be **stopped** — it locks the datadir (and, with `--keep-blocks`, additionally opens `chaindata/` exclusively). The v3.4.0 release notes put the rebase itself at roughly ten seconds; the cost that follows is re-acquiring whatever was in `chaindata/`.

llms-full.txt:2073

  • The claim that this command is safe to rerun after an interruption is unsafe. applyStepRebasePlan writes the new settings only after renaming every file, while collectRenameList rescales every filename it currently finds. If the process stops midway, the old step size remains recorded and a rerun rescales already-renamed files a second time, potentially corrupting ranges. Remove this guarantee and direct operators to restore/recover a partially renamed datadir.
The command prints the full rename and delete plan and asks for confirmation (`Proceed with these changes? [y/N]`) before touching anything, and exits as a no-op if the datadir is already at the requested step size. It is **destructive and has no built-in undo**, but it is safe to re-run after an interruption. The node must be **stopped** — it locks the datadir (and, with `--keep-blocks`, additionally opens `chaindata/` exclusively). The v3.4.0 release notes put the rebase itself at roughly ten seconds; the cost that follows is re-acquiring whatever was in `chaindata/`.

docs/site/static/llms-full.txt:2073

  • The claim that this command is safe to rerun after an interruption is unsafe. applyStepRebasePlan writes the new settings only after renaming every file, while collectRenameList rescales every filename it currently finds. If the process stops midway, the old step size remains recorded and a rerun rescales already-renamed files a second time, potentially corrupting ranges. Remove this guarantee and direct operators to restore/recover a partially renamed datadir.
The command prints the full rename and delete plan and asks for confirmation (`Proceed with these changes? [y/N]`) before touching anything, and exits as a no-op if the datadir is already at the requested step size. It is **destructive and has no built-in undo**, but it is safe to re-run after an interruption. The node must be **stopped** — it locks the datadir (and, with `--keep-blocks`, additionally opens `chaindata/` exclusively). The v3.4.0 release notes put the rebase itself at roughly ten seconds; the cost that follows is re-acquiring whatever was in `chaindata/`.

docs/site/docs/fundamentals/snapshots-management.mdx:185

  • With --nibbles.v2 omitted, the converter targets V1; it does not unconditionally leave the existing key encoding unchanged. A file currently using V2 will be rewritten to V1, so this sentence understates what the example command can change.
Both encoding flags default to `false`, and `false` is the target state that matches what v3.6 writes — so the command above, with no encoding flags, converts referenced files to plain and leaves the key encoding alone.

llms-full.txt:2032

  • With --nibbles.v2 omitted, the converter targets V1; it does not unconditionally leave the existing key encoding unchanged. A file currently using V2 will be rewritten to V1, so this sentence understates what the example command can change.
Both encoding flags default to `false`, and `false` is the target state that matches what v3.6 writes — so the command above, with no encoding flags, converts referenced files to plain and leaves the key encoding alone.

docs/site/static/llms-full.txt:2032

  • With --nibbles.v2 omitted, the converter targets V1; it does not unconditionally leave the existing key encoding unchanged. A file currently using V2 will be rewritten to V1, so this sentence understates what the example command can change.
Both encoding flags default to `false`, and `false` is the target state that matches what v3.6 writes — so the command above, with no encoding flags, converts referenced files to plain and leaves the key encoding alone.

llms-full.txt:2049

  • Conversion is driven by both target axes, not only whether values are plain. A plain file using V2 keys is also rewritten when the documented default --nibbles.v2=false target is used, so this cost estimate should refer to files not already in the full plain/V1 target state.
**What does it cost?** Every commitment `.kv` file that is not already plain is rewritten and its accessors (`.kvi` / `.bt` / `.kvei`) rebuilt. Converted output is written into `snapshots/rebuild/domain/` while the originals are still in place, so at peak you need free space for a second copy of the commitment domain — use the `commitment` row of `erigon seg du --datadir <path> -v` to size it. Plain output is larger than the referenced input.

docs/site/static/llms-full.txt:2049

  • Conversion is driven by both target axes, not only whether values are plain. A plain file using V2 keys is also rewritten when the documented default --nibbles.v2=false target is used, so this cost estimate should refer to files not already in the full plain/V1 target state.
**What does it cost?** Every commitment `.kv` file that is not already plain is rewritten and its accessors (`.kvi` / `.bt` / `.kvei`) rebuilt. Converted output is written into `snapshots/rebuild/domain/` while the originals are still in place, so at peak you need free space for a second copy of the commitment domain — use the `commitment` row of `erigon seg du --datadir <path> -v` to size it. Plain output is larger than the referenced input.

docs/site/docs/fundamentals/snapshots-management.mdx:202

  • Conversion is driven by both target axes, not only whether values are plain. A plain file using V2 keys is also rewritten when the documented default --nibbles.v2=false target is used, so this cost estimate should refer to files not already in the full plain/V1 target state.
**What does it cost?** Every commitment `.kv` file that is not already plain is rewritten and its accessors (`.kvi` / `.bt` / `.kvei`) rebuilt. Converted output is written into `snapshots/rebuild/domain/` while the originals are still in place, so at peak you need free space for a second copy of the commitment domain — use the `commitment` row of `erigon seg du --datadir <path> -v` to size it. Plain output is larger than the referenced input.

Comment thread docs/site/docs/fundamentals/snapshots-management.mdx Outdated
Comment thread llms-full.txt Outdated
Comment thread docs/site/static/llms-full.txt Outdated
Comment thread docs/site/docs/fundamentals/configuring-erigon.mdx Outdated
Comment thread llms-full.txt Outdated
Comment thread docs/site/static/llms-full.txt Outdated
Comment thread docs/site/docs/fundamentals/snapshots-management.mdx Outdated

`seg reset` is **destructive and not reversible**: at the default `--local=true` it deletes `chaindata/`, the Heimdall / Polygon-bridge DBs, and any locally generated snapshot files. Independently of `--local` — that is, even with `--local=false` — it also deletes any `.torrent` file whose infohash does not match the preverified list, **together with the snapshot data file that torrent vouched for** (a stale local build is unverifiable, so the downloader must re-fetch the canonical copy). All other preverified snapshots already on disk are kept and corrected by the downloader on the next start. The node must be **stopped** — the command takes an exclusive lock on the datadir.

Cost: whatever the downloader has to re-fetch, plus rebuilding state on the next start. Run `--dry-run` first, and back up the datadir.

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This bounds the cost by the downloader, but the slow tail isn't the downloader.

Blocks freeze into torrent-restorable segments only up to keep = 1024 behind tip (block_snapshots.go:215), and the published preverified manifest can lag the live tip by much more. Anything synced past what the manifest covers exists only in chaindata/, which --local=true deletes unconditionally. That range comes back through Caplin's per-block beacon backfill, not a segment fetch — orders of magnitude slower, and on a node far enough behind it takes a second restart before execution clears ExecutionStatusTooFarAway.

Worth one line so the reader sizes the risk by how far past the manifest they are, rather than by download time.

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Fixed in 64fcab3, and the framing is now yours: size the cost by how far the node is past the published manifest, not by download time.

One correction to the citation, though — I checked block_snapshots.go before writing it. Freezing is bounded by CanRetire, which uses dbg.MaxReorgDepth (96) plus 1000-block segment alignment. The 1,024 in CanDeleteTo governs when chaindata copies of already-frozen blocks may be deleted, not when blocks freeze. So the paragraph states the manifest-coverage point without attributing it to that function.

Kept: only the manifest-covered range is torrent-restorable, --local=true removes both chaindata/ and locally built segments, and the tail returns via Caplin per-block backfill rather than a segment fetch. I softened the second-restart claim to "may need more than one restart" since I could not confirm the ExecutionStatusTooFarAway sequence from source alone — it rests on your live test.

@bloxster bloxster changed the title docs(site): document the v3.6 plain commitment format and its migration paths [r3.6] docs(site): document the v3.6 plain commitment format and its migration paths Aug 27, 2026
… reset cost

- Referenced encoding needs both a sub-v2.2 stamp and a >=2-step range
  (CommitmentBranchReferenced), so a single-step v2.1 file is plain.
- --chain is a root-level flag and resolves in either position.
- Size seg reset by how far the node is past the preverified manifest:
  segments freeze only to 1,024 behind tip, and the tail returns through
  Caplin per-block backfill, not a segment fetch.
@bloxster
bloxster enabled auto-merge August 27, 2026 15:36
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants