Commit the round reference time in the SMT leaf value

[ristik]

Background

Yellowpaper commit 407b07f ("commit reference time in the SMT leaf value") changes the Unicity Service SMT leaf value from the transaction hash alone to H(txhash, τ), where τ is the reference time of the round the request was validated in (UC.IR.t).

Predicate evaluation already takes τ as an argument, but τ was recoverable only from the inclusion proof, as UC.IR.t. The SMT is append-only, so a leaf can be certified afresh against any later root, and a later proof carries a later round's IR.t. Reference time was therefore a property of the proof rather than of the leaf, and re-presenting a leaf changed the predicate evaluation outcome. Binding τ into the leaf value fixes the value the transition was validated under, for any proof of that leaf.

τ cannot go into txhash: it is chosen by the Aggregator after the sender has signed. The leaf value is the first field written once τ is known, and it is the one field re-presentation preserves verbatim.

Affected yellowpaper sections: platform.tex (Unicity Service Request, request validation, SMT leaf structure, inclusion proof), execution-layer.tex (mint/transfer transaction structure and verification, sec:time-extraction, Unicity Service processing), appendix-hashtrees.tex (ZK-compressed consistency proof).

Normative encodings

These must be byte-identical across aggregator-go, rugregator, bft-core, and the three SDKs.

Symbol	Meaning	Type
τ	Round reference time: UC.InputRecord.Timestamp of the round the leaf was certified in. BFT Core requires it to equal the previous round's Unicity Seal timestamp.	uint64, Unix seconds
τ_Q	Exclusive request timeout, chosen by the sender	uint64, Unix seconds

Transaction hash. τ_Q is the last element of the transaction's own deterministic CBOR array, so

txhash = SHA-256( <transaction CBOR, ending in τ_Q> )

is unchanged in form and is exactly appendix-token.tex's SHA-256(CBOR(…, τ_Q)). Keeping τ_Q inside the transaction rather than beside it (as the appendix's four-element certified transaction does) means the request, the certified transaction and the hash preimage cannot disagree about it, and the unlock script signs it along with the rest of the transaction.

SMT leaf value.

v = SHA-256( CBOR([ txhash, τ ]) )     // 32 raw bytes

txhash enters as a CBOR byte string of the raw 32-byte digest (no algorithm-id prefix), τ as a CBOR unsigned integer. This is the same construction already used for StateID and for the signature preimage.

Shared test vector:

txhash = 000102030405060708090a0b0c0d0e0f101112131415161718191a1b1c1d1e1f
τ      = 1755000000
preimage = 825820000102…1e1f 1a689b2cc0
v      = 0235bd52cfa10c9785dfa01942bc396f201fe715dbc3896ee117a97e895e1e36

Certification request. CertificationData carries τ_Q between the transaction hash and the witness, matching Q = (ρ, sthash, txhash, τ_Q, u). The aggregator never sees the transaction, so it needs the value explicitly to enforce expiry.

Inclusion proof. Carries τ. A verifier cannot recover it from the certificate chain, because an aggregator serves proofs against the current certified root rather than the one the leaf was created under.

Certified transaction. [ transaction, τ, inclusionProof ]. τ_Q is already inside element 0.

Admission rule. A request may be inserted only in a round whose reference time satisfies τ < τ_Q.

Scope in aggregator-go

The SMT leaf value becomes H(txhash, τ), where τ is the reference time of the round the leaf is inserted in. τ is the value this node already puts into IR.Timestamp (the previous UC's Unicity Seal timestamp, internal/bft/client.go:buildCertificationInputRecord), so no new source of truth is introduced. The leaf value and IR.Timestamp of a round must be derived from the same pinned τ, or BFT Core will not reproduce the root.

Round must pin τ

τ is known when a round starts (it is the seal timestamp of the UC that triggered the start) but leaves are currently inserted before it is recorded anywhere:

• internal/round/round_manager.go — Round needs a pinned ReferenceTime, set by StartNewRound / StartNewRoundWithSnapshot from the UC that starts the round.
• internal/bft/client.go — buildCertificationInputRecord must use the round's pinned τ rather than re-reading the latest UC at proposal time, so a UC arriving mid-round cannot make IR.Timestamp disagree with the leaves already inserted. When the pinned τ and the latest UC disagree, the round has to be discarded and re-collected rather than proposed.
• internal/round/precollector.go — the precollector inserts the next round's leaves into a forked snapshot while the current round is still awaiting its UC, i.e. before that round's τ exists. Leaf materialisation has to move to the point where τ is known; precollection of the requests themselves can stay.

Leaf value derivation

• internal/models/certification_request.go — LeafValue() takes τ and returns SHA-256(CBOR([txhash, τ])).
• internal/round/leaf_add.go — commitmentLeafInput threads the round's τ.
• internal/round/recovery.go:506 — recovery recomputes leaves for a round; it must use that round's τ, read back from the stored block's UC.

Serving τ to clients

A verifier cannot recover τ from the proof: this node serves inclusion proofs against the current certified root, so UC.IR.t of a served proof is the latest round's reference time, not the one the leaf was created under. That divergence is exactly what the yellowpaper change fixes, and it means τ has to be returned explicitly.

• Store the round's τ on models.CertificationRequest and models.AggregatorRecord (BSON included).
• pkg/api — InclusionProofV2 carries τ and GetInclusionProofV2 fills it from the record.
• pkg/api/types.go:InclusionProofV2.Verify and internal/proofverify/local.go derive the leaf value as SHA-256(CBOR([txhash, τ])) instead of using the raw transaction hash.

Notes

• Non-backward-compatible and state-resetting. There is no mainnet; no migration path is required, but a stored SMT from before the change cannot be reused.
• Cross-implementation agreement is what makes this safe: rugregator, bft-core (branch l1) and the three SDKs land the same encoding, and the change is switched on across the deployment in one coordinated step.

Acceptance criteria

• A round's leaf values and its IR.Timestamp provably come from one pinned τ.
• Inclusion proofs carry τ, and local proof verification reconstructs the root from H(txhash, τ).
• Recovery and restart reproduce identical roots for already-certified rounds.
• Unit tests pin the leaf-value encoding against a fixed vector shared with the other implementations.

---

Cross-repository change. The same encoding lands in aggregator-go, rugregator, bft-core (branch l1, consistency proof only) and the three state transition SDKs, and is switched on across the deployment in one coordinated step. Cross-implementation test vectors are part of the work.

Companion issues

Repository	Reference time in the leaf value	Request timeout
aggregator-go	#176	#177
rugregator	ristik/rugregator#3	ristik/rugregator#4
bft-core (branch l1)	unicitynetwork/bft-core#23 (RSMT), unicitynetwork/bft-core#24 (ZK)	out of scope
state-transition-sdk-js	unicitynetwork/state-transition-sdk-js#144	unicitynetwork/state-transition-sdk-js#145
state-transition-sdk-java	unicitynetwork/state-transition-sdk-java#81	unicitynetwork/state-transition-sdk-java#82
state-transition-sdk-rust	unicitynetwork/state-transition-sdk-rust#16	unicitynetwork/state-transition-sdk-rust#17

[b3y0urs3lf]

QA closing report — Issue #176

Resolved by PR #178 (implementation), with follow-ups PR #180 (cleanups + metric) and PR #182 (proof binding + referenceTime on block records). All 4 acceptance criteria verified via a mix of aggregator-side QA and SDK-side BDD suite.

Setup tested

• Branch: ha-bft-hardening HEAD (aa87ccc), which has main + PR Bind service time and enforce request timeouts #178/Cleanups from reviewing #178: single-pass CBOR decode, TTL default, drop metric #180/fix(api): bind inclusion proofs to the certified shard and network; return referenceTime on block records #182 merged
• Topology: 2-shard, 2-replica HA bft-sharding stack via make docker-run-bft-sh-ha-clean
• SMT backend: RocksDB, layout string yellowpaper-rsmt-sha256-v6a-be
• DB precondition: clean DB (PR Bind service time and enforce request timeouts #178 marked as state-resetting per issue text — no migration required, no mainnet existed)
• Host: Intel i7-11375H (4C/8T), 62 GB RAM, ext4 on LVM, Ubuntu 24.04.4, Docker 29.5.0

What was asked (Issue #176)

• Change SMT leaf value from H(txhash) to H(txhash, τ), where τ is the round's reference time (UC.InputRecord.Timestamp)
• Bind τ into the leaf so re-presented leaves cannot carry a different reference time
• Extend InclusionProof to carry τ; verifiers must reconstruct root from SHA-256(CBOR([txhash, τ]))
• Recovery must reproduce identical roots for certified rounds using stored τ
• Match the shared cross-implementation test vector: v = 0235bd52cfa10c9785dfa01942bc396f201fe715dbc3896ee117a97e895e1e36 (for txhash = 0001…1e1f and τ = 1755000000)
• Add unit tests validating the encoding against that vector
• Non-backward-compatible, state-resetting change (no migration required; no mainnet existed)

What was delivered — with code evidence

• Leaf value function — pkg/api/leaf_value.go (new file, ~23 lines): LeafValue(txhash, referenceTime) = SHA-256(CBOR([txhash, referenceTime]))
• Per-round τ pinning — internal/round/round_manager.go stores pinned τ when the round starts; used consistently throughout the round's lifecycle rather than reading the latest UC timestamp each time
• Leaf construction — internal/models/certification_request.go LeafValue() method incorporates the pinned τ from the round context
• BFT client — internal/bft/client.go submits certification using the round's pinned τ, not the latest UC timestamp
• Recovery / replay — internal/round/recovery.go recomputes leaves using the stored block's τ, not current wall clock; disk-SMT bounded replay from internal/round/disk_smt_startup.go walks finalized blocks and reconstructs using stored τ per block
• Wire format — pkg/api/types.go:153 — InclusionProofV2 is a tagged 5-element CBOR array: [version, certificationData, referenceTime, certificateBytes, unicityCertificate]. referenceTime at position 2 is new
• Wire spec doc — docs/inclusion-proof-wire.md (new file, pkg/api/types.go and inclusion_cert.go cite it as normative)
• Unit test with cross-impl vector — pkg/api/leaf_value_test.go reproduces 0235bd52cfa10c9785dfa01942bc396f201fe715dbc3896ee117a97e895e1e36 for the pinned inputs

Delivered but not requested in Issue #176

Not defects — design decisions and hardening that landed in PR #178 and its follow-ups but are not called out in the issue text:

• New dropped-commitments metric (PR Cleanups from reviewing #178: single-pass CBOR decode, TTL default, drop metric #180): aggregator_commitments_dropped_total{reason="duplicate|expired|rejected"} — observability for commits that never reach a block
• CBOR decode performance (PR Cleanups from reviewing #178: single-pass CBOR decode, TTL default, drop metric #180): CertificationData.UnmarshalCBOR refactored from a two-pass to a single-pass decode; roughly 5700 ns → 3000 ns per commitment (~50% improvement)
• DEFAULT_REQUEST_TTL consolidated (PR Cleanups from reviewing #178: single-pass CBOR decode, TTL default, drop metric #180): single-source constant instead of two divergent copies (env default + fallback)
• commitmentLeafInput → materializeCommitmentLeaf rename (PR Cleanups from reviewing #178: single-pass CBOR decode, TTL default, drop metric #180): reflects that the function has a persistence side effect, not just returning a value
• AggregatorRecord.ReferenceTime field (PR fix(api): bind inclusion proofs to the certified shard and network; return referenceTime on block records #182): block records now carry the pinned τ per record, so consumers can reconstruct leaf value from the record alone. Not stated in Issue Commit the round reference time in the SMT leaf value #176 but required by the reconstruction contract
• Independent BE-reference test — internal/smt/be_reference_anchor_test.go (from PR Switch SMT to big-endian bit ordering #170) still runs post-Bind service time and enforce request timeouts #178 and validates that leaf reconstruction under the new formula matches an independent BE reference implementation
• Wire spec documentation — docs/inclusion-proof-wire.md is new; the issue asked for an implementation, this shipped a normative spec doc alongside it

QA coverage — how each item was verified

Aggregator-side tests (this QA track):

• Empty-root sanity — verified that a freshly-cleaned cluster reports diskRootHash = 0000…0000 on all 4 replicas at startup (BE genesis matches yellowpaper ⊥). Confirms the empty-root semantic under the new leaf formula
• Reference-time replay integrity — killed shard0 leader with committed blocks in place, restarted it, watched the disk-SMT bounded replay reconstruct 24 blocks in ~110 ms with matching root. Because replay computes leaves from stored records using their block's stored τ, a formula regression would produce a root mismatch and the replica would refuse to start. It came up clean. This is the critical PR Bind service time and enforce request timeouts #178 correctness gate for the new formula
• HA leader kill under live SDK load — killed shard0 leader mid-stream during SDK sequential BDD run. All submitted commitments either verified or resubmitted cleanly. No verify failures, no duplicate finalized rows. Confirms new leaf formula holds under HA failover
• Cross-shard correlated crash — killed both shard leaders simultaneously. Both standby's promoted in <30 s. Block production resumed on both shards. Regression envelope check for PR Bind service time and enforce request timeouts #178's changes to round/finalize paths
• Metrics visibility — aggregator_commitments_dropped_total{reason=...} counter increments correctly on the live cluster

SDK-side (state-transition-sdk-js) tests:

• S1 canary vector — hermetic reproduction of 0235bd52…166ede from the SDK's independent calculateLeafValue(txhash, τ) implementation. Cross-implementation byte-for-byte match
• S2 live-cluster 100 mints — SDK submitted 100 real commitments against this PR's running cluster, reconstructed each returned root via its own SHA-256(CBOR([txhash, τ])) walk, and compared with the aggregator's returned block root. 100/100 byte-matched. This transitively pins CBOR framing, hash formula, and SMT interior-node computation against Go across a realistic sample
• S7 InclusionProofV2 wire fixture — aggregator QA captured a live proof (docs/interop-fixtures/inclusion-proof-v2-with-reftime.json) and the SDK's hermetic decoder round-trips it, reading referenceTime at CBOR position 2. Cross-language handshake artifact for Java + Rust SDKs
• Full SDK BDD sweep — 461/461 scenarios green in ~85 min against a cluster running the merged code

Verdict

Every acceptance criterion in Issue #176 has code evidence at specific file:line locations, and every one is verified either by direct aggregator QA testing, SDK-side testing, or both. The cross-implementation vector 0235bd52…166ede reproduces on Go, JS SDK, and the aggregator's live wire path.

No data loss, no root divergence, no verification failures observed under any tested scenario.