Switch SMT root/inclusion proof hashing from version 3o to 6a

[ristik]

Update the SMT implementation from the current 3o-style node commitment to the v6a construction from rsmt6a.py.

Scope:

• No backwards compatibility required; this is a coordinated pre-production rollover.
• Leaf hash stays unchanged: it still commits to the full key and value.
• Inclusion proof shape stays unchanged: bitmap/depth set + sibling hashes.
• Internal node hash changes to also commit to the node's canonical region/prefix.

Implementation notes:

• Old 3o node commitment was effectively:
  • H_node(depth, left_hash, right_hash)
• New v6a node commitment is:
  • H_node(depth, region, left_hash, right_hash)
• region is the canonical prefix for that node at depth, using this repo's existing key bit order, endian-ness, and packing conventions.
• Use the existing Go encoding rules.
• During inclusion proof verification, no extra proof field is needed. Recompute region = prefix(proven_key, depth) at each sibling-combine step.
• Ensure tree construction stores/derives the canonical region for every internal node.

Acceptance:

• Update existing root and inclusion-proof tests to v6a hashes.
• Add shared interop vectors generated with this implementation and verify vectors from the JS, Java, and Rust SDKs.
• Include cases for empty tree, one leaf, shallow split, deep split, and multi-leaf tree.

[MastaP]

https://github.com/ristik/ndsmt-experiments/blob/main/rsmt6a.py

[ristik]

unicity-yellowpaper.pdf - Appendix C3

[b3y0urs3lf]

QA closing report — PR #168 / Issue #167

Ready to merge. Two independent test tracks green: aggregator-side QA against a live cluster and SDK-side BDD suite via state-transition-sdk-js.

Setup tested

• Branch: ha-bft-hardening HEAD with feat/smt-v6a-node-hash (commits c811807 + 3f4a6f6) merged in for QA.
• Topology: 2-shard, 2-replica HA bft-sharding stack via make docker-run-bft-sh-ha-clean (bft-sharding-ha-compose.yml). 10 containers.
• SMT backend: RocksDB. Confirmed via /metrics — aggregator_smt_disk_proof_snapshot_hold_duration_seconds and aggregator_smt_disk_proof_snapshots_active are only present when
  the disk-SMT path is engaged.
• DB precondition: clean DB, no legacy stateId_1 unique index. Layout string on disk = yellowpaper-rsmt-sha256-v6a.
• Host: Intel i7-11375H (4C/8T), 62 GB RAM, ext4 on LVM, Ubuntu 24.04.4, kernel 6.17, Docker 29.5.0.

What Issue #167 asked

• Switch internal node hash from 3o-style H_node(depth, L, R) to v6a-style H(0x01 || depth || region_32B || L || R).
• Leaf hash unchanged.
• Inclusion-proof wire shape unchanged (bitmap/depth set + siblings).
• Region derived from inserted keys at creation.
• Region reconstructed from NodeKey + compressed path at load, validated by key.DepthBits() + path.Len() == depth.
• Empty-tree root = all-zero.
• TreeLayout bumped; older data requires rebuild.
• Interop vectors: empty, one-leaf, shallow split, deep split, multi-leaf. Verify from JS + Java + Rust SDKs.

Reviewer flag from PR comments (gemini-code-assist bot): shallow-key masking bug in RegionFromKeyBytes — fixed by commit 3f4a6f6.

What code delivered — with file references

• Ask 1 — node hash formula. In-memory internal/smt/smt.go:426-446 (NodeBranch.calculateHash). Disk internal/smt/disk/hash.go:53-62 (HashNode). Legacy path
  pkg/api/smt.go:148-158 (via RegionFromPathBits).
• Ask 2 — leaf hash unchanged. Still H(0x00 || key || value) at internal/smt/smt.go:337-342, internal/smt/disk/hash.go:39-47, pkg/api/inclusion_cert.go:108-117. Diff
  confirms zero changes to these three sites.
• Ask 3 — wire shape unchanged. InclusionCert at pkg/api/inclusion_cert.go:56-92 is still Bitmap[32] || Siblings[n][32]; MarshalBinary/UnmarshalBinary byte-for-byte
  compatible. What changed is the verifier's per-step hash formula, not the wire.
• Ask 4 — region at creation. In-memory: internal/smt/smt.go:822-828, 842-849. Disk build: internal/smt/disk/tree.go:408, 474, 587, 681.
• Ask 5 — region reconstruction + depth invariant. internal/smt/disk/serde.go:105-117 (RegionForNode); UnmarshalInternal:159-166 explicitly asserts
  key.DepthBits()+path.Len() != int(depth) and errors out.
• Ask 6 — empty root = all zero. internal/smt/disk/hash.go:81-83 returns Hash{}. In-memory short-circuit at internal/smt/smt.go:416-423. Unit test
  TestV6AInterop_EmptyTree pins the hex.
• Ask 7 — TreeLayout bump + old-data rejection. internal/smt/disk/rocksstore/store.go:67 sets TreeLayout = "yellowpaper-rsmt-sha256-v6a". Open-time rejection at :946-951
  with error unsupported disk SMT tree layout %q, want %q.
• Ask 8 — interop vectors. internal/smt/v6a_interop_vectors_test.go — all 5 required scenarios (empty at line 78, one-leaf at 86, shallow split at 98, deep split bit-255 at
  112, multi-leaf at 133) plus JS SDK parity 6-key vector at line 57 pinning cd23fc12654.... No Go-side vectors for Java or Rust — PR description says those verify externally.
• Reviewer's masking bug — fixed at pkg/api/inclusion_cert.go:271-275 with the maskByte < byteLen guard.

How QA verified — aggregator-side direct tests

Four tests driven directly against the running 2-shard cluster on the merged HEAD.

• T1 Empty-root sanity — passed. All 4 replicas started with diskRootHash: 0000000000000000000000000000000000000000000000000000000000000000 in their startup logs. First three
  finalized blocks all have root 0000.... aggregator_smt_nodes_persisted_total = 0. v6a genesis matches the spec ⊥.
• T4 HA leader kill + state-ID continuity under load — passed. cmd/performance-test at ~1000 rps × 2 shards for 60s, docker kill aggregator-bft-shard0 at T+15s. Result:
  32,013 / 32,013 commitments verified with valid inclusion proofs, 0 failures, 0 STATE_ID_EXISTS. Same envelope as PR Harden disk SMT failover recovery #161 QA — v6a code holds HA failover identically.
• T5 Outage-window measurement (T6 forcing function). Baseline (LOCK_TTL_SECONDS=10, HAProxy check inter 2s fastinter 1s rise 2): outage window = 11.76s. Tuned
  (LOCK_TTL_SECONDS=3, HAProxy check inter 500ms fastinter 250ms rise 1): outage window = 5.49s. 54% reduction. Remaining ~5.5s is dominantly BFT partition round + LOCK expiry
  wait — further reduction to ~3s possible with LOCK_TTL_SECONDS=1 but at the cost of flappy failovers.
• T6 Cross-shard correlated crash — passed. Killed both leaders (aggregator-bft-shard0-b and aggregator-bft-shard1-b) simultaneously. Both standby's promoted to leader in
  under 30s. Block production resumed on both shards.

How SDK verified — external independent track

SDK BDD suite delivered under @disk-smt-coverage @v6a-hash @cross-impl-parity:

• 14/14 hermetic scenarios green — V1-V6, V8-V12 covering empty tree, one-leaf, shallow split at bit 0, deep split at bit 255, shallow-key regression against commit 3f4a6f6,
  sequential mint burst, state-transition mint→transfer→verify, depth-boundary sweep, multi-shard consistency, and 4-level split-mint recursion.
• V5 — 6-key JS SDK parity vector, hermetic reproduction of cd23fc1265484a7173323cd862b85a61796b8e0af31149944a828e6c1734b846 byte-for-byte from
  internal/smt/v6a_interop_vectors_test.go:57-75.
• V4 + V10 — deep-split with Scenario Outline at bits {8, 16, 32, 64, 128, 240, 255}. Wider coverage than the aggregator's own interop test set.
• V7 — live-cluster hash reconstruction, 100 real mints. SDK submitted 100 commitments against the running aggregator, fetched each inclusion proof, reconstructed the
  aggregator-returned root via InclusionCertificate.calculateRoot. 100 / 100 byte-matched over 4m59s wall. This is the interop-contract proof at production surface with real BFT
  signing — end-to-end SDK↔aggregator hash agreement.

What was delivered in code but NOT documented in Issue #167, PR description, or comments

Not bugs, but design decisions that landed silently and belong in code comments or a dev-facing design note:

• NodeBranch gains a Region []byte field at internal/smt/smt.go:276. The in-memory tree now carries 32 bytes per internal node. The PR description talks only about
  disk-side region derivation at load; it never says the in-memory tree keeps region resident. Memory overhead scales with internal-node count. Not tested by QA (perf territory) — dev
  owns any memory-budget follow-up.
• Parent-mode populate() region math at internal/smt/smt.go:132-158. Recursive region threading through fully-populated parent-tree construction, with a specific
  right-subtree bit-0-set trick. No interop vector covers parent-mode. Not tested by QA (no public API exposes parent-mode).
• Two new exported public API helpers — RegionFromKeyBytes at pkg/api/inclusion_cert.go:255-277 and RegionFromPathBits at pkg/api/smt.go:191-205. New public surface;
  downstream Go consumers can now depend on their contract. Not called out.
• Runtime hash-length panic guard in constructors at internal/smt/smt.go:69-75, 94-98. NewSparseMerkleTree/NewChildSparseMerkleTree panic if hashLen(algorithm) != smtCachedHashBytes. Defensive check, unrelated to v6a semantics but landed in the same commit.
• Empty-root change disclosed in PR description but NOT in Issue Switch SMT root/inclusion proof hashing from version 3o to 6a #167 text. Anyone reading only the issue won't know empty-tree semantics changed. QA test T1 above pins this externally.
• In-package API signature changes — NewInternal(path, depth, region, L, R) at internal/smt/disk/node.go:52-75 (updated across ~7 call sites) and decodeBranch(key, encoded)
  at internal/smt/disk/persist/tree.go:1794 (updated across 6 callers). Package-internal but non-trivial refactor.

What QA deliberately did NOT test

• Interop vector via aggregator JSON-RPC API. submit_commitment requires cryptographic signatures and cannot accept raw (key, value) tuples, so the 6-key fixture and the
  deep-split bit-255 case can't be reproduced through the production API from the aggregator side. Covered externally by SDK V5 (hermetic) and V7 (100/100 live). Not duplicated.
• Deep-split bit-255 via aggregator JSON-RPC API. Same crypto-signature blocker. Covered by SDK V4 + V10.
• TreeLayout-version rejection via poking RocksDB directly. The code-level assertion at store.go:946-951 is already tested by the PR's own tests; the RocksDB-poking exercise is
  over-engineered for QA black-box scope.
• Memory footprint at 60M-leaf steady state. Perf territory, dev owns.
• Java + Rust SDK cross-implementation vectors. Separate SDK rounds when those SDKs land v6a.
• Parent-mode region math. No public API exposes parent-mode from the aggregator — audit finding is dev-facing.

Verdict

• Every asked behavior in Issue Switch SMT root/inclusion proof hashing from version 3o to 6a #167 has code evidence at specific file:line locations.
• Wire format is genuinely unchanged.
• The reviewer-flagged masking bug is fixed and covered by the new interop tests.
• 14 hermetic SDK scenarios + V7 live 100/100 byte-matched hash reconstruction against this cluster is the definitive interop-contract proof.
• Aggregator-side QA 4/4 green including the T6 outage-window forcing function.

Recommended for merge. The undisclosed-extras list above is documentation debt for a follow-up commit, not a merge blocker.