From 234365ffb6860951d16f6860846b7e926a3d138c Mon Sep 17 00:00:00 2001 From: Rehan Khan Date: Sat, 22 Aug 2026 23:54:46 +0500 Subject: [PATCH] docs: record OpenTimestamps lifecycle decision --- docs/adr/0002-opentimestamps-lifecycle.md | 349 ++++++++++++++++++++++ 1 file changed, 349 insertions(+) create mode 100644 docs/adr/0002-opentimestamps-lifecycle.md diff --git a/docs/adr/0002-opentimestamps-lifecycle.md b/docs/adr/0002-opentimestamps-lifecycle.md new file mode 100644 index 0000000..165ab3a --- /dev/null +++ b/docs/adr/0002-opentimestamps-lifecycle.md @@ -0,0 +1,349 @@ +# ADR 0002: OpenTimestamps lifecycle + +- Status: Proposed +- Date: 2026-08-22 +- Decision owner: VESTRIX maintainer + +This ADR defines the boundary between the immutable forensic chain, asynchronous +OpenTimestamps submission, proof upgrading, and independent Bitcoin +verification. It does not claim that the design is implemented. The current +Python production backend and full Rust anchor verdict remain intentionally +fail-closed until an implementation conforming to this ADR is reviewed. + +## 1. Digest-boundary handling + +### Context + +The forensic chain already computes a 32-byte SHA-256 `record_hash` for its +selected tip. That digest is the value the current Rust proof inspector expects +inside the detached OpenTimestamps proof. + +The `ots stamp FILE` command hashes `FILE`; it does not accept an already +computed digest for submission. Passing a temporary file containing the 32 +digest bytes would therefore anchor `SHA256(record_hash_bytes)`, not +`record_hash`. Hiding that extra layer would make the Python submitter and Rust +verifier disagree about the evidentiary boundary. + +### Decision + +Use the lower-level `opentimestamps` Python library directly to construct a +`DetachedTimestampFile` whose SHA-256 file digest is exactly the existing chain +tip `record_hash`. Build the timestamp graph from that digest and retain the +client's normal privacy protection: append a fresh random nonce and hash the +result before submitting the calendar commitment. + +Use the official `opentimestamps-client` implementation for calendar submission +and proof upgrading, behind a narrow VESTRIX adapter. Do not invoke `ots stamp` +on digest bytes and do not define a VESTRIX-specific extra-hash proof format. + +### Consequences + +- The detached proof remains directly bound to the existing `record_hash`. +- Python and Rust can share the published OpenTimestamps wire format without + sharing implementation code. +- Calendar servers see a nonce-protected commitment rather than the raw chain + tip digest. +- The adapter must isolate the package's CLI-oriented internals and translate + exceptions or process-exit behavior into explicit worker failures. +- Tests must assert that the detached proof's embedded digest is byte-for-byte + equal to the selected chain-tip `record_hash`. + +### Rejected Alternatives + +- **Write the digest bytes to a temporary file and run `ots stamp`.** Rejected + because it silently adds a SHA-256 layer and changes what is being attested. +- **Document and teach the Rust verifier about the extra hash layer.** Rejected + because it creates a VESTRIX-specific semantic exception with no security or + interoperability benefit. +- **Build the raw calendar protocol in VESTRIX.** Rejected because the official + libraries already implement submission, serialization, and upgrading; a new + protocol client would add security-sensitive code unnecessarily. + +## 2. Submission architecture + +### Context + +Calendar requests have network latency and can fail. Bitcoin confirmation and +proof upgrading can take hours. Collector ingestion must not wait for either. +The signed JSONL chain is immutable evidence and cannot also serve as a mutable +job-status store. + +### Decision + +Implement anchoring as a one-shot worker process separate from the collector. +An operating-system scheduler starts it once when the service deployment comes +up and every 15 minutes afterward. A systemd timer is the reference Linux +deployment; cron or an equivalent scheduler may invoke the same command. + +Each scheduled run: + +1. briefly acquires the existing chain lock and snapshots the current + `(sequence, record_hash)` tip; +2. inserts that tip as `queued` if it is not already tracked; +3. processes due submission, upgrade, and verification work; and +4. exits. + +If no records have been appended since the most recent tracked tip, the run is +a no-op. Calendar and Bitcoin operations occur after releasing the chain lock. + +Persist lifecycle state in a separate SQLite database under the evidence +directory, with detached `.ots` proof files stored beside it. SQLite is chosen +because it is transactional and is provided by the Python standard library. +The state database records at least: + +- chain sequence and tip hash; +- current proof state and proof path; +- attempt count, last attempt, next attempt, and last error; +- the state from which a failed operation should resume; and +- verified Bitcoin height, block hash, verification time, and confirmation + count when available. + +Enforce uniqueness on `(sequence, tip_hash)`. Write new or upgraded proofs to a +temporary file, flush and `fsync` them, and atomically replace the sidecar proof +before committing the corresponding state transition. Never append lifecycle +metadata to, or rewrite, the forensic JSONL chain. + +### Consequences + +- Collector acceptance and hash-chain append remain independent of calendar or + Bitcoin availability. +- The maximum normal unanchored interval is approximately 15 minutes, plus + scheduler and submission delay. +- A stopped worker does not corrupt the chain, but monitoring must alert that + external timestamp coverage is stale. +- The SQLite database is operational metadata, not primary evidence. A lost + database can be reconstructed by inspecting retained proofs and chain tips; + an `independently_verified` claim must still be reproducible from the chain, + proof, and Bitcoin data. +- Deployment gains one scheduled command and one evidence-sidecar directory, + not another long-running service. + +### Rejected Alternatives + +- **Run submission in the collector event loop.** Rejected because calendar + latency or failure would affect ingestion availability. +- **Run an in-process collector background task.** Rejected because it couples + anchoring lifetime, failure handling, and upgrades to collector restarts. +- **Create a permanent anchoring daemon.** Rejected because a scheduled + one-shot worker provides the required isolation and retry behavior with less + operational state. +- **Store lifecycle transitions in the signed JSONL chain.** Rejected because + pending and retry state is mutable operational data and would blur or require + rewriting the evidentiary record. +- **Use one mutable JSON status file.** Rejected because crash-safe concurrent + transitions and recovery are already provided by standard-library SQLite. + +## 3. Proof state machine + +### Context + +Calendar acceptance is not Bitcoin confirmation, and a proof containing a +Bitcoin attestation has not yet been independently checked against Bitcoin's +active chain. A single `anchored` flag would overstate assurance and make retry +behavior ambiguous. + +### Decision + +Use these explicit states: + +| State | Meaning | Transition trigger | +|---|---|---| +| `queued` | A unique chain tip has been durably recorded, but no calendar submission proof has been durably stored. | Created from a new scheduled tip snapshot. Moves to `submitted_pending` after a valid pending proof is atomically stored, or directly to `upgraded_unverified` if the stored response already contains a Bitcoin block-header attestation. | +| `submitted_pending` | Calendar submission succeeded and a durable proof exists, but it contains no Bitcoin block-header attestation yet. | Remains pending when the calendar reports "not ready". Moves to `upgraded_unverified` when an upgraded proof with a Bitcoin attestation is atomically stored. | +| `upgraded_unverified` | The proof reaches a Bitcoin block-header attestation, but no independent active-chain check has succeeded. | Moves to `independently_verified` only after the Rust verifier succeeds against the configured Bitcoin Core node and the confirmation policy is met. | +| `independently_verified` | The Rust verifier authenticated the selected chain tip, replayed the proof, matched its commitment to a block header on Bitcoin Core's active chain, and observed the required confirmations. | Terminal for normal processing of that proof. A later explicit revalidation may report a deep-reorganization or evidence-integrity failure. | +| `retry_failed` | The most recent submission, upgrade, storage, or independent-verification attempt failed. | Stores `retry_from`, failure class, error, attempt count, and next attempt. A due transient retry returns to `retry_from`; an integrity failure disables automatic retry and requires operator review. | + +Calendar `not ready` is normal pending state, not a failure. Network timeouts, +calendar quorum failures, temporary Bitcoin RPC unavailability, and temporary +filesystem or lock failures are retryable. Malformed proofs, digest mismatches, +and a commitment that does not match the active-chain block header are integrity +failures: they enter `retry_failed`, produce an alert, and receive no automatic +retry until reviewed. + +Use bounded exponential backoff with jitter for transient failures. A successful +later operation clears the prior error but retains attempt history. State labels +are operational summaries; only re-running verification over the retained +artifacts establishes the evidentiary result. + +### Consequences + +- Operators and documentation can distinguish calendar acceptance, Bitcoin + attestation availability, and independent verification. +- `submitted_pending` may legitimately last hours without blocking ingestion or + being reported as failure. +- Proof corruption and digest divergence fail closed and cannot be hidden by + repeated automatic submission. +- The worker must make every transition idempotent and recover correctly when a + process stops between proof persistence and database commit. + +### Rejected Alternatives + +- **A Boolean `anchored` field.** Rejected because it conflates at least three + materially different assurance levels. +- **Treat calendar submission as independent verification.** Rejected because a + pending calendar receipt is not a Bitcoin timestamp. +- **Treat all upgrade misses as failures.** Rejected because confirmation delay + is expected protocol behavior. +- **Retry integrity failures indefinitely.** Rejected because repeated network + work cannot repair a malformed or mismatched evidentiary artifact. + +## 4. Rust verifier scope for v0.9 + +### Context + +The current Rust verifier authenticates forensic chains and partially inspects +OpenTimestamps proofs. It intentionally returns `anchor verification +incomplete` after reaching a Bitcoin attestation because the proof contains a +height, not authoritative evidence that the corresponding header belongs to +Bitcoin's active best chain. + +The project roadmap assigns the independent verifier and OpenTimestamps +anchoring to v0.8-v0.9, with the acceptance condition that a third party can +verify a log without running VESTRIX. Deferring the final Bitcoin check while +calling v0.9 complete would contradict that condition. + +### Decision + +v0.9 requires full independent anchor verification in `verifier-cli` using a +Bitcoin Core RPC dependency. A successful Rust anchor verdict must: + +1. authenticate the complete forensic chain with the supplied Ed25519 public + key and identify the selected tip; +2. verify that the detached proof begins at that exact tip digest; +3. independently parse and replay every proof operation needed by proofs + emitted by the pinned Python client; +4. ask Bitcoin Core for the active-chain block hash at each candidate attested + height and retrieve the corresponding header; +5. compare the proof-derived commitment with the header's Merkle root; and +6. require the configured confirmation threshold before returning success. + +The default proposed confirmation threshold is six. Bitcoin RPC unavailability, +unsupported proof operations, insufficient confirmations, malformed proof data, +digest mismatch, Merkle-root mismatch, or a height not present on the active +chain all fail closed. There is no public block-explorer fallback. + +The Rust verifier must not call the Python submitter or `ots verify`; its proof +logic remains a separate implementation. Bitcoin Core is the verifier's +operator-controlled consensus data source, independent of the collector, +forensic logger, Python OpenTimestamps adapter, and public calendars. A pruned +Bitcoin Core node is acceptable because verification needs headers and active +chain lookup rather than historical transaction bodies. + +The current `anchor verification incomplete` behavior remains correct until all +of these conditions are implemented and tested. It must not be replaced with a +success result in stages. + +### Consequences + +- v0.9 can satisfy the repository's existing independent-verification + milestone rather than carrying a known gap into the credibility-focused v1.0 + phase. +- Independent verification requires access to a synchronized Bitcoin Core node + and introduces RPC configuration and an additional Rust dependency. +- A verifier can be distributed without Python or trust in VESTRIX's calendar + submission path. +- CI can test the RPC boundary deterministically with regtest fixtures, but a + release-evidence verification must use a synchronized independently operated + Bitcoin mainnet node. +- The verifier only needs to accept valid operations emitted by the pinned + supported client; unsupported valid OTS variants continue to fail closed + until deliberately added. + +### Rejected Alternatives + +- **Keep anchor verification incomplete throughout v0.9 and defer Bitcoin + headers to v1.0+.** Rejected because it conflicts with the stated v0.9 + acceptance condition and leaves the project's independent timestamp claim + unverifiable. +- **Delegate verification to Python `ots verify`.** Rejected because Python + submission and verification would share one implementation and trust + boundary. +- **Use a public blockchain explorer.** Rejected because the explorer's answer + would become an unaudited trusted assertion. +- **Accept an operator-supplied standalone block header.** Rejected because one + header does not establish membership in Bitcoin's active best chain. +- **Add the existing Rust `opentimestamps` crate and assume verification is + complete.** Rejected because parsing and replaying a proof does not supply + authoritative best-chain data or confirmation policy. + +## 5. Pinned dependency versions + +### Context + +OpenTimestamps proof construction and calendar communication are +security-sensitive and have previously remained stubbed because no reviewed +client version was pinned. Version claims in this ADR must be tied to a dated +package-index check. + +PyPI was checked directly on 2026-08-22: + +- [`opentimestamps-client`](https://pypi.org/project/opentimestamps-client/) has + latest release `0.7.2`, published 2024-12-31. +- Its current PyPI metadata requires + `opentimestamps>=0.4.0,<0.5.0`. +- [`opentimestamps`](https://pypi.org/project/opentimestamps/) has latest release + `0.4.5`, published 2023-01-25. +- Both projects are classified Beta on PyPI and use the LGPLv3-or-later license. + +### Decision + +The implementation must declare: + +```text +opentimestamps-client==0.7.2 +opentimestamps>=0.4.0,<0.5.0 +``` + +The reproducible environment or lock artifact used for release testing must +record the resolved core package exactly; with the package index state checked +above, that resolution is `opentimestamps==0.4.5`. + +Before implementation is merged, review the selected distributions' hashes, +licenses, transitive dependencies, calendar defaults, timeout/quorum behavior, +and the exact internal functions wrapped by VESTRIX. Re-check PyPI at that time; +newer releases require a separate compatibility and security review rather than +an automatic upgrade. + +### Consequences + +- Submission and upgrade behavior is reproducible for the first implementation. +- VESTRIX accepts maintenance responsibility for a narrow adapter around a Beta, + CLI-oriented package. +- Security updates are deliberate changes requiring proof-vector, pending-state, + upgrade, and cross-language verification tests. +- This ADR records versions only; dependency files remain unchanged until the + implementation task is approved. + +### Rejected Alternatives + +- **Install unversioned latest releases.** Rejected because behavior could + change without review and invalidate cross-language proof assumptions. +- **Pin directly to a Git commit.** Rejected because released PyPI artifacts are + easier to reproduce, hash, cache, and audit. +- **Depend only on `opentimestamps-client` and leave the core resolution + unconstrained.** Rejected because VESTRIX imports lower-level proof objects + directly and must make that compatibility range explicit. +- **Vendor the Python packages immediately.** Rejected because there is no + demonstrated need to own a fork; reviewed upstream distributions are the + smaller trust and maintenance surface. + +## Open questions for maintainer sign-off + +1. Is a 15-minute checkpoint interval an acceptable evidence-recovery window, + or must deployments use a shorter interval? +2. Is six Bitcoin confirmations the required default for + `independently_verified`, and should deployments be permitted to increase but + not decrease it? +3. Which public calendars and submission quorum should be approved instead of + inheriting client defaults implicitly? +4. What retention and backup policy applies to pending proofs, upgraded proofs, + and the reconstructible SQLite lifecycle database? +5. Should `retry_failed` with `failure_class = integrity` remain the only + operator-review state, or should implementation introduce a separate terminal + `invalid` state? +6. Which Bitcoin Core RPC authentication mechanism and Rust RPC crate/version + should the implementation standardize on? RPC credentials must not be passed + on a process command line. +7. Has the maintainer accepted the LGPLv3-or-later and Beta-status implications + of the proposed Python dependencies?