perf(wal): derive operation contexts from validated committed prefixes

[flyingrobots]

Merged into main as 017a60f1 after current-head independent review and hosted gates.

Summary

Healthy operation-context queries repeatedly decoded the committed WAL prefix. This change derives a context index from validated history, advances it only after acknowledged appends, and reconciles uncertain writes before checking identity or constructing another transaction. Read-only fallback resolves retained context without restoring stale writer permission.

Scope

• Runtime code
• Tests
• Canonical documentation and changelog

Links

Closes #777. This extracts the independently mergeable cache outcome from #729, on merged prerequisite #727. It deliberately excludes the retained-strand outcome of #728. After this replacement lands, #729 can close as superseded; its original history and historical benchmark evidence remain preserved.

Validation

• Fresh Docker RED on committed baseline e5b3543c: eight healthy capture/read/bind/retry cycles add 48 recovery calls and decode 376 frames, failing the unchanged-work assertion.
• GREEN at 53579e1b: 126 WAL tests (one existing ignored), 42 operation-pipeline, 44 host-loop and thirteen CLI tests pass, plus formatting and strict supported-feature host Clippy. Fault cases cover append-frame, flush-commit and commit-marker-synced boundaries, original request retry, changed-input refusal and reopening.
• Current candidate b5743754 merges landed Retain observation-bound executable requests across host loss #727 without conflicts; its tracked source tree is unchanged from the tested cache candidate. This is source-bound earlier execution, not a falsely relabeled clean final-head run.
• These counters establish decoded-history work, not current throughput, CPU time or physical I/O. The timing table on original Avoid repeated WAL reconstruction in trusted host contexts #729 remains historical and has not been reprofiled.

Documentation accuracy

The canonical hosting document now describes acknowledged-only derived index publication, uncertain read/write posture and optional diagnostic counters. README/GUIDE authority boundaries are unchanged. The WAL remains authority; the prefix digest is not fencing. This changes no WAL encoding or durability, installs no retained strands and adopts no production Keep backend.

Checklist

• Current-head hosted CI green (fmt, Clippy, tests, rustdoc, audit/deny and applicable gates)
• One coherent, independently mergeable outcome
• Final exact-head independent review approved and live threads reconciled

[chatgpt-codex-connector]

You have reached your Codex usage limits for code reviews. You can see your limits in the Codex usage dashboard.
To continue using code reviews, add credits to your account and enable them for code reviews in your settings.

[coderabbitai]

Warning

Review limit reached

You've used all free OSS reviews for now. Wait for the free limit to reset to keep reviewing this public repository.

Next included review available in 32 minutes.

Check out review usage here.

View limit details

Limit details: You’ve used the included review currently available.

Learn how review limits work.

Review configuration:

⚙️ Run configuration

• Configuration used: Repository: flyingrobots/echo/.coderabbit.yaml
• Review profile: ASSERTIVE
• Plan: Advanced
• Run ID: fca24a2d-bcb2-43ce-9a0a-72b81c92e1f2

📥 Commits

Reviewing files that changed from the base of the PR and between 2c976ef and b574375.

📒 Files selected for processing (7)

• CHANGELOG.md
• crates/warp-core/src/causal_wal.rs
• crates/warp-core/src/trusted_runtime_host.rs
• crates/warp-core/src/trusted_runtime_host/observed_context.rs
• crates/warp-core/tests/trusted_runtime_host_loop_tests.rs
• docs/architecture/application-contract-hosting.md
• xtask/src/run_edict_operation/session.rs

• Autopilot · Keep fixing CodeRabbit findings and required CI, and resolving merge conflicts

---

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

• X
• Mastodon
• Reddit
• LinkedIn

_{Comment @coderabbitai help to get the list of available commands.}

[flyingrobots]

Current-head independent verdict: APPROVE b574375473e3738e5496ff27e8f48315cedc81df. The final refresh supersedes the preparatory validation request preserved below. Hosted CI remains a separate gate.

Independent adversarial review preparation: validated-prefix context cache

Inspected clean candidate 53579e1b0475070de584ae0c8eb43af7cf68909b on observation prerequisite 21968dcd3a22898f81b2fab7a926d5b2a2f7c8d2. This prepares the cache-only replacement of #729. Final integration head, current CI and final review remain pending; this is not approval of the old stacked PR or unpublished replacement.

No new verified blocking code defect found in the complete seven-file delta.

Mandatory Verification Checklist

• Entire delta inspected: causal_wal diagnostic counters, TrustedRuntimeWal index/state transitions, observed_context validation and borrowed lookup, three fault/performance regressions, one optional stderr profile route and canonical docs/changelog. Diff334 additions/73 removals matches the extracted original cache slice. No retained-strand implementation or prerequisite is present.
• Initialization trusted_runtime_host.rs:2502–2529: same validated recovered report builds writer cursor, context index, durable submissions and evidence catalog before becoming available. Index is bound to previous committed transaction digest. This digest identifies a validated prefix; it is not a writer fencing token or a replacement causal authority.
• Incremental validation observed_context.rs:54–115: from_recovery and apply_frames use the same context parser. Only context-kind frames affect the derived map. Schema, bounded identity, original observation, invocation, semantic digest and duplicate IDs are checked. An invalid frame cannot become published cache state merely because it arrived through append rather than recovery.
• Healthy reads observed_context.rs:120–133: matching prefix lends immutable contexts through Cow::Borrowed. Missing/mismatched binding performs read-only recovery into owned contexts. This read does not restore write permission or publish a cache bound to a stale writer cursor. Returned observations/request bytes are cloned only at the existing public API boundary.
• Context-write reconciliation observed_context.rs:135–150,178–190,320–335: uncertain prefix is reconciled before observation/request identity checks and before building new context transaction. Durable-but-reported-failed request is therefore found before a duplicate or replacement append; immutable observation replacement still refuses.
• Append trusted_runtime_host.rs:3290–3330: derives next coordinates, removes candidate from circulation, checks prefix, applies new frames, calls storage, and publishes the candidate only after successful acknowledgement. Type/context/prefix/storage errors cannot publish unacknowledged candidate. Pre-storage coordinate overflow cannot append. Any storage uncertainty leaves None, blocking stale-cursor transactions.
• Shared error/recovery callers trusted_runtime_host.rs:2871–2957: existing submission acceptance, Tick commit, runtime control, package installation and causal-anchor after-error checks call the same refresh_cursor_from_store_for_writer. :2831–2868 clears index first, recovers writable prefix, constructs complete cursor/contexts, and reinstalls binding only after validation and cursor updates. A refresh error leaves index unavailable. This applies to ordinary transactions as well as context frames, preventing a parallel append path from keeping an incorrectly healthy index.
• In-memory rollback trusted_runtime_host.rs:2589: includes operation_context_index alongside cloned in-memory WAL and existing cursor/index evidence. Restored memory state therefore carries its matching derived-prefix index; filesystem durability is not faked with an in-memory rollback snapshot.
• Errors: new OperationContext preserves typed parser failure; UncertainWriterPrefix is an explicit refusal. Context parser retains previously existing string error surface; no new success-on-error, nested wrapper recovery loop or public backend authority introduced. Poisoned append attempts are reconciled rather than automatically assuming storage did not commit.
• Diagnostic counters causal_wal.rs:6165–6191: thread-local cumulative pair counts calls before decoding, including failures, and decoded frame count after segment decoding succeeds. They do not count physical reads, CPU time or materialization cost. No WAL frame/schema/durability change. Thread-local state affects diagnostics only, not deterministic admission or causal identities.
• Optional CLI profile session.rs:120–123: presence of ECHO_WAL_PROFILE emits cumulative pair on stderr after the existing response flush; stdout protocol unchanged. Documented example=1 enables it; any environment presence currently enables it. No documented claim that=0 disables profiling.
• Numeric/current doc claims: causal-history authority, acknowledged-only derived cache publication, owned fallback reads, write reconciliation, immutable retry, process-local counters and absence of CPU/I/O measurement all match code. Existing observation4,096-byte/16-slot and context1,024/4,096 limits remain unchanged; cache does not enlarge grants or replace native provenance.

Extraction, historical review and evidence

• Extracted coherent nonmerge commits: e5b3543 (regressions/counters),db11e07e (cache implementation),53579e1b (equivalent predicate simplification), each based on observation21968. Existing original Avoid repeated WAL reconstruction in trusted host contexts #729 merge commits containing retained-strand ancestry were intentionally not transplanted; no broken intermediate mainline requirement is invented. Final merge onto landed Retain observation-bound executable requests across host loss #727 must be inspected separately.
• Fully paginated original Avoid repeated WAL reconstruction in trusted host contexts #729: zero threads,zero reviews,four global comments, no further pages. Comments consist of provider skip/rate-limit and review activity rather than effective approval. Original headd8f8821d targets retained-strand branch; its stale checks cannot approve this independently extracted candidate.
• Original benchmark table is explicitly pinned to historicalea1f03e1 and synthetic retained procedure with uncontrolled OS caches. Its13.264/7.074-second timing figures and403/14recoveries belong to that historical experiment, not this new head. I inspected the caveats; did not independently rerun or authenticate all historical benchmark artifacts. No new speedup/throughput figure is claimed in changed canonical docs.
• Actual RED landing-cache-red.log binds regression baselinee5b3543c: healthy workload counters grow from(3,0) to(51,376), failing expected unchanged work. This demonstrates repeated recovery, not a measured timing benefit.
• GREEN log currently reports126 WAL tests/one existing ignored,42 executable pipeline,44 host-loop and13 CLI tests passing. New host-loop covers healthy observation/request reads/writes/retries without scans; AppendFrame/FlushCommit/CommitMarkerSynced uncertainty; unchanged request retry and changed-input refusal after durable reported failure and reopen. Exact runtime state-machine behavior is witnessed by parent-run tests, not reviewer execution. Final formatting/Clippy/result and final main integration still require confirmation.
• Root/budget validation remains the existing parent guard:20GiB build,4GiB data,128MiB logs and50GiB free floors; reviewer did not use worker,cache or service. Final receipt/source manifests must bind the integrated candidate before approval.

Judgment and coverage limits

Read-only source/Git/API/log inspection executed. No tests, Docker, scripts, compiler or runtime run by reviewer. No physical power-loss, retained-strand acceptance, authenticated external service or current throughput claim established. Current-head integration and final hosted gates are unfinished, so this preparatory report does not approve a merge.

REQUEST CHANGES — final main integration, completed validation/source binding and current-head review pending.

Final exact-head review refresh: PR #778

The preparatory verdict is superseded by this refresh. Reviewed clean and live head b574375473e3738e5496ff27e8f48315cedc81df, targeting main 2c976ef55245df121e21b6182b209ae60cf4a16d, replacement PR778 closing issue777. No remaining verified blocking defect found.

• Complete seven-file production/error/counter/doc checklist above remains applicable to unchanged bytes. The new merge b574 has parents53579e1b0475070de584ae0c8eb43af7cf68909b and landed observation main2c976ef55245df121e21b6182b209ae60cf4a16d. First-parent tree diff is empty: no conflict resolution or production change. Against main it is exactly the seven-file cache slice,334 additions/73 removals. Observation/main invariants and source-function publication remain intact; retained strands remain excluded.
• GREEN completed:126WAL tests/one existing ignored,42 pipeline,44 host-loop and13CLI passed. Formatting and strict supported-feature host Clippy completed, result exit0. Healthy context queries avoid further decoded-history work; fault controls cover precommit and durable-but-reported-failed contexts/requests, immutable retry, changed semantic refusal and reopen. No throughput benefit is inferred from these counters.
• Independently SHA-256 checked all1,013 GREEN manifest files against final clean b574 tree: zero mismatches. Manifest names53579e1b, the actually tested clean source commit. Empty integration merge preserves the exact tree; evidence is source-bound earlier execution, not falsely called a fresh b574 run.
• Final guard receipt inspected: build14,353,929,547bytes<20GiB; data4,144,256,331<4GiB; logs23,207,342<128MiB; host/VM free bytes exceed50GiB. Reviewer ran no workloads or shared workers.
• Live PR metadata confirms b574 head and2c976ef5 target, mergeable. Full new PR connections retrieved: zero threads,zero reviews,two conversation comments, all pageInfo exhausted. Comments are quota/rate-limit notices, not an effective provider review or additional finding. Old729 has zero threads/reviews and four paginated comments, and remains historical traceability rather than authority for this replacement.
• New PRbody counters are checked against actual RED:51 minus3 =48 added recovery calls;376 decoded frames beyond initial0. Its counts and source-coordinate distinction match GREEN/manifests. Canonical docs state derived acknowledged-only index and explicit uncertain posture; optional counters are logical decoded work, not CPU/physical IO. Original historical timing table is not repeated as current evidence. No WAL encoding/durability, retained-strand, production Keep or new fencing claim.
• Current CI at review remains running: six successes,31in progress,two queued. This independent approval does not open the merge gate or permit bypass. Recheck current head, fully paginated fresh discussions, required checks and protections immediately before merge; close old729 only after replacement integration.

Ran: read-only source/Git/hash/API inspection. Read: completed GREEN/RED logs, source manifest, launch/result receipts and exact PRbody. Inferred: empty-merge preservation and state-machine compatibility from inspected code. No tests, compiler/runtime, Docker or repository scripts executed by reviewer. No physical power-loss or benchmark replication evidence established.

Exact-head verdict: APPROVE b574375473e3738e5496ff27e8f48315cedc81df. Hosted CI and final live merge gates pending.

APPROVE