WAL durability modes: none, deferred, strict; batched submission (tracking) #787

Activity

1. flyingrobots

   added

   enhancementNew feature or requestNew feature or request

   runtimeRuntime coreRuntime core

   coreEcho Core TechnologiesEcho Core Technologies

   needs-witnessMethod closeout state needs-witness.Method closeout state needs-witness.

   on Oct 13, 2026
2. flyingrobots commented on Oct 13, 2026

   

   flyingrobots

   on Oct 13, 2026

   OwnerAuthor

   More actions

   Review corrections to the design (PR #788, now a draft pending owner acceptance; commit fcb881e).

   • A failed final sync acknowledges nothing. A readable record is not a durable one.
   • Deferred acknowledgements are bound to writer epoch and transaction identity, not to a log position alone.
   • Records after the durable position may be lost; recovery can legitimately rebuild beyond it.
   • A full deferred queue makes the caller wait and never refuses, so writer timing cannot decide admission.
   • The one-writer law applies to modes that write a WAL.

   Existing behavior worth its own fix. After a failed append or sync of a single submission, submit_intent_with_runtime_wal_ack re-reads the log (recover_filesystem_submission_acceptance_after_error) and treats a readable acceptance record as accepted. If the failure was the sync, that record is not known to be durable. This predates this work. It is not changed by #791, which returns the sync error for a batch.
3. flyingrobots commented on Oct 13, 2026

   

   flyingrobots

   on Oct 13, 2026

   OwnerAuthor

   More actions

   Unit 2 (tick record cost): findings. Measured in the guarded worker, release build, WAL on tmpfs, 32 Actions per tick.

   • Fixed in the PR that references this issue: the retained state delta was built and digested once per Action; it is now built once per commit. Tick p50 6.19 ms to 4.69 ms.
   • Tested and rejected: writing a transaction's frames with one append instead of one per frame. No gain (4.69 ms to 4.71 ms).
   • Where the remaining WAL time goes (about 3.4 ms of the 4.7 ms): the tick transaction is 66 frames and about 125 KB, roughly 3.9 KB per Action. Store append about 1.1 ms; evidence catalog update about 0.5 ms; per-Action outcome records about 0.3 ms; the rest is building and hashing frames.

   So the remaining cost is data volume and hashing. Options that need a decision, not just code: retain fewer bytes per Action (what each retained Action outcome must carry), share repeated identities across the frames of one transaction, or hash each payload fewer times on the append path. This matches the study feedback's observation that WAL bytes per fact are large relative to the payload.
4. flyingrobots commented on Oct 13, 2026

   

   flyingrobots

   on Oct 13, 2026

   OwnerAuthor

   More actions

   Pre-existing weaknesses found while reviewing #791 (not introduced or worsened by it; each needs its own change):

   1. TrustedRuntimeWal::from_config runs writable recovery, which can truncate a torn tail, before the writer lock is taken. Two hosts opening one root at the same time could both truncate. Acknowledge a batch of submissions after one WAL sync #791 syncs existing segments at open, which removes the unsynced-tail hazard but not this ordering.
   2. refresh_cursor_from_store_for_writer records last_good_commit for a failed evidence-catalog rebuild from the cursor value before the refresh, not after it.
   3. The single-submission path still treats a readable acceptance as accepted after a failed append or sync (noted earlier on this issue). The batched path now fails closed instead; the two should be brought into line, in favour of failing closed.
   4. Hosted CI did not run trusted_runtime_host_loop_tests at all, or causal_wal_hardening_tests with fault injection, before Acknowledge a batch of submissions after one WAL sync #791 added both steps.
5. flyingrobots commented on Oct 13, 2026

   

   flyingrobots

   on Oct 13, 2026

   OwnerAuthor

   More actions

   Native macOS measurements (2026-10-13). Apple M5 Pro, internal SSD, main at 3056a9d9, release profile, Homebrew Rust 1.98.1 (not the pinned 1.96.0), single runs. Storage sync here is F_FULLFSYNC, about 3.7 ms.

   Scene test: N entities (nodes with one attachment each); each frame submits K compare-and-set Actions on K distinct entities as one batch and ticks until all are committed (64 Actions per head per tick, so K/64 ticks). 30 frames, frame p50 / p95 in ms. "In-memory WAL" builds every WAL record and never touches storage; it is the nearest available stand-in for the caller-side cost of the proposed deferred mode, which is not implemented.

   Entities	Updates per frame	No WAL	In-memory WAL	Strict disk WAL
   10,000	64	2.6 / 2.8	12.9 / 13.8	29.9 / 31.8
   10,000	128	3.3 / 3.7	17.1 / 19.8	46.7 / 47.9
   10,000	256	6.7 / 7.1	37.4 / 48.3	86.6 / 91.4
   10,000	512	14.0 / 14.7	101 / 136	171 / 177
   10,000	1,024	29.5 / 30.6	288 / 441	not run
   100,000	64	3.2 / 3.4	13.3 / 14.0	31.7 / 32.5
   100,000	512	18.7 / 19.9	96 / 141	174 / 188
   100,000	1,024	37.8 / 39.8	299 / 461	not run

   Updates per frame that stay at or above 30 fps (p95 ≤ 33.3 ms): about 1,000 without a WAL (about 850 at 100,000 entities); between 128 and 256 with the in-memory WAL; 64 with the strict disk WAL, with no margin.

   Findings:

   1. Scene size barely matters now (1,000 to 100,000 entities changes a 64-update frame from 2.5 to 3.2 ms). The cost is per update.
   2. The WAL record is the cost. Without a WAL an update costs about 40 µs. With records built and not synced it costs about 200 µs at 64 updates. Storage sync is a minority of the strict frame.
   3. The in-memory WAL cost per update grows with the amount written: 200 µs at 64 updates per frame, 290 µs at 1,000, and p95 well above p50 within a 30-frame run. Something in the record path scales with accumulated log or with batch count per frame. Not yet located.
   4. Standard frames (K Actions, one tick), strict disk, native: single submissions 8.8 ms (K=1), 42.8 ms (K=8), 152.8 ms (K=32); batched 8.8, 14.5, 18.0 ms. Batching is what makes 8 to 32 Actions per frame possible at all on this disk.
6. flyingrobots

   added a commit that references this issue on Oct 13, 2026

   Fix: state queue waits, mid-batch failure, batch limits and fault pos…

   ...

   c873201
7. flyingrobots commented on Oct 13, 2026

   

   flyingrobots

   on Oct 13, 2026

   OwnerAuthor

   More actions

   Gap found in review of #788: submit_intents_with_runtime_wal_ack has no limit on envelope count or encoded bytes, so a caller can submit an arbitrarily large batch against a fixed frame budget. Proposed in the design page: a host-set batch limit, checked against the whole batch before the first append, with a typed over-budget refusal.
8. flyingrobots commented on Oct 13, 2026

   

   flyingrobots

   on Oct 13, 2026

   Last edited by flyingrobots

   OwnerAuthor

   More actions

   Proposal: deferred durability mode (revision 2, not accepted)

   This proposal moved here from PR #788. docs/architecture holds current architectural truth, so an unaccepted design lives in this issue until the owner accepts it. PR #788 (merged) documents only the batching that shipped in #791, in docs/architecture/batched-strict-submission.md. Revision 2 applies review guidance relayed by the owner on 2026-10-13 to four review findings. That guidance is input to the proposal; the proposal as a whole is not accepted.

   The idea in one line

   The trade is when Echo promises durability. The meaning of "accepted" does not change with the host's performance setting.

   Two separate guarantees

   Guarantee	Meaning	Same in every mode?
   Accepted	The runtime committed the input to its ordered history.	Yes
   Durable	Recovery will rebuild the record after a crash.	No; this is what a mode chooses

   A caller learns about durability from a separate value, never from the word "accepted":

   • a durable frontier: every record at or before it will be rebuilt by recovery; and
   • a durability handle per acknowledgement, which completes when the frontier passes that record, or fails.

   Modes

   Mode	Accepted when	Durable when	Caller waits for
   None	Committed to ordered history	Never	Nothing
   Deferred	Committed to ordered history, with queue capacity already reserved	The writer's next group sync covers it	Nothing by default; queue capacity only if the caller asks to wait
   Strict	Committed to ordered history	Before the call returns (filesystem store)	One storage sync

   Strict with the in-memory store is not durable; see docs/architecture/batched-strict-submission.md.

   Deferred mode

   A writer thread inside the Echo process owns the WAL store and its writer-epoch lease. The runtime hands it records through a bounded queue and does not touch storage itself. The writer appends in acceptance order, syncs in groups, and publishes the durable frontier after each sync.

   An acknowledgement names its record by identity (writer epoch and transaction) and carries its log position. Identity matters because a position can be reused after a crash loses a record: a durability handle is satisfied only by the record with the same identity.

   Admission and queue pressure (replaces the earlier timeout rule)

   Review finding: the earlier text let elapsed queue waiting make the host declare a failure, which let writer speed decide what was admitted.

   1. Reserve before accepting. The runtime reserves queue capacity for a record before it commits the acceptance. An accepted input therefore always has a place in the queue.
   2. No capacity is not a refusal. When no capacity is free, the submission call either waits or returns a distinct status: not admitted, retryable. Nothing is committed to history and nothing is recorded as a refusal of the input. The caller may submit the same envelope again.
   3. Writer slowness never becomes a lawful refusal. Refusals describe the input against the runtime's laws. Queue delay is not one of those laws. No timeout on queue waiting converts a slow writer into a failed one.
   4. Once accepted, an input cannot disappear because the writer fell behind.

   Consequence for determinism: writer timing can change when an input is admitted and therefore which tick first sees it, exactly as host scheduling already can. It cannot change what the runtime decides about an input it has been offered and admitted. Two runs offered the same admitted inputs in the same order produce the same history.

   Decided by the owner on 2026-10-13 (the owner explicitly adopted review guidance to this effect). The deferred mode as a whole remains proposed and not accepted:

   • "Not admitted, retryable" records nothing in authoritative history. It is a structured submission result. Optional diagnostic telemetry may record the attempt, but telemetry must not advance history or imply acceptance.
   • The call returns that status immediately by default. Waiting for capacity is an explicit caller option, and the caller can cancel the wait at any point before acceptance. Hidden waiting would defeat the frame-budget motivation; a host chooses its own scheduling policy.

   Writer failure (new)

   Review finding: the earlier text did not say what happens to a caller already holding a deferred acknowledgement when the writer fails.

   A writer failure is an actual failure signal from the writer (an append or sync error, or the writer thread ending), never an inference from delay.

   When the writer fails:

   1. Pending durability handles fail explicitly. Each outstanding handle for a record after the last confirmed durable frontier completes with a fault that names the writer epoch and that frontier. No handle waits forever.
   2. Acceptance stops. The runtime accepts no further input under that writer until recovery or an explicit mode transition. A caller refused for that reason receives an obstruction that names the fault. The fault and the obstruction stay separate typed values, as the Obstructions topic requires.
   3. Accepted inputs stay accepted in memory. Their durability is unconfirmed, not denied: the operating system may have written a further prefix. A caller treats a record after the frontier as indeterminate until recovery reports it.
   4. The last confirmed durable frontier is reported.
   5. No silent fallback. The runtime does not continue as mode None.

   Crash behaviour

   A crash in deferred mode may lose records after the durable frontier. Recovery rebuilds a valid prefix of what the runtime accepted, at least up to the frontier and possibly beyond it. It never rebuilds a gap. Recovery reports a lost acknowledgement as lost.

   Laws

   1. Accepted has one meaning in every mode. Durability is reported separately.
   2. Durability reports are observations. The durable frontier and writer timing never influence what a tick computes or whether an offered input is lawful.
   3. Order is fixed at acceptance. The writer never reorders, merges or drops.
   4. Back-pressure is not refusal. Lack of queue capacity delays admission or returns "not admitted, retryable". It never records a refusal.
   5. A writer failure is observable. Handles fail, acceptance stops, the frontier is reported.
   6. The log is an ordinary Echo WAL. One recovery path reads every mode.
   7. One writer. Deferred mode moves the single writer to a thread.
   8. A mode is declared in host configuration and is visible in evidence. A receipt produced without a WAL, or before its record was durable, does not claim strict durability.

   Also proposed (strict mode)

   • A host-set batch limit on envelopes and encoded bytes, checked before the first append; a batch over the limit is refused whole with a typed over-budget result. No limit exists today.
   • Strict single submission should stop treating a readable record as accepted after a failed sync, as the batch path already does.

   Alternative not chosen

   An external WAL process on a port. Same speed as deferred mode, but it adds a wire protocol, a second process and a second holder of storage authority, and its failure cases differ in both directions (host dies after hand-off; writer dies while host runs on). That is a larger contract than this design needs.

   Open decisions (owner)

   1. Accept or reject this proposal.
   2. Whether a strict frame should cover its submissions and its tick with one sync instead of two.
   3. Whether "not admitted, retryable" leaves evidence. Decided: nothing in authoritative history; optional telemetry only.
   4. Whether a host may change mode while a runtime is live, and what evidence the change leaves.
   5. Whether the submission call waits by default. Decided: return immediately; waiting is opt-in and cancellable.
9. flyingrobots

   added a commit that references this issue on Oct 13, 2026

   Merge main; document shipped batching only

   ...

   fc0c8b1