From 6fbb7c5ac60c2327339c828adf7c89a22882c58a Mon Sep 17 00:00:00 2001 From: James Ross Date: Mon, 12 Oct 2026 20:01:52 -0700 Subject: [PATCH 1/7] docs: propose WAL durability modes and batched submission Record a proposed, not yet implemented design for when Echo waits for storage: three host-chosen modes (none, deferred, strict), batched strict submission that appends ordinary intake transactions and syncs once, and a deferred mode with an in-process writer thread, a bounded queue and an observable durable position. State the laws that keep it honest: acknowledgements state their posture, durability reports never influence admission or ticks, nothing is dropped silently, and every mode writes an ordinary Echo WAL. Record what the runtime does today and name issue #787 as the owner of implementation state. Co-Authored-By: Claude Opus 5.5 --- docs/README.md | 1 + docs/architecture/wal-durability-modes.md | 144 ++++++++++++++++++++++ docs/topics/WAL.md | 4 + 3 files changed, 149 insertions(+) create mode 100644 docs/architecture/wal-durability-modes.md diff --git a/docs/README.md b/docs/README.md index ecbcc53d..33f0b928 100644 --- a/docs/README.md +++ b/docs/README.md @@ -20,6 +20,7 @@ causal history. Git history is the archive; GitHub owns live work and status. - [There Is No Graph](architecture/there-is-no-graph.md) - [Sliced readings and copy-on-write strands](architecture/sliced-readings-and-copy-on-write-strands.md) (accepted design, not implemented) +- [WAL durability modes and batched submission](architecture/wal-durability-modes.md) (proposed design, not implemented) - [Continuum transport](architecture/continuum-transport.md) - [Echo optics adapter notes](architecture/echo-optics-adapter-notes.md) - [Echo × Keep physical content boundary](architecture/echo-keep-physical-content-boundary.md) diff --git a/docs/architecture/wal-durability-modes.md b/docs/architecture/wal-durability-modes.md new file mode 100644 index 00000000..36964635 --- /dev/null +++ b/docs/architecture/wal-durability-modes.md @@ -0,0 +1,144 @@ + + + +# WAL Durability Modes and Batched Submission + +- **Status:** Proposed design. **Not implemented.** The section + [Current implementation](#current-implementation) states what the runtime + does today. +- **Proposal date:** 2026-10-12 +- **Refines:** [WAL](../topics/WAL.md) +- **Related:** [Sliced readings and copy-on-write strands](sliced-readings-and-copy-on-write-strands.md), + [Fixed timestep invariant](../invariants/FIXED-TIMESTEP.md) + +## Problem + +A host that runs Echo at a fixed frame rate has a fixed time for each frame. +At 60 frames per second that time is 16.67 ms. + +A strict WAL acknowledgement waits for a storage sync. Today each submission +is one transaction with one sync, and each scheduler pass is another. A frame +with many submissions therefore waits for many syncs, one after another. The +count of syncs, not the work of the tick, decides whether the frame fits. + +The rule of the WAL topic stays in force: Echo may only claim what its WAL can +recover. This page changes when Echo waits and what an acknowledgement says. +It does not change what Echo may claim. + +## Decision + +A host chooses one of three durability modes for a runtime. + +| Mode | An acknowledgement means | The caller waits for | +| -------- | ---------------------------------------------------------- | -------------------- | +| None | Accepted in memory. Nothing is recoverable. | Nothing | +| Deferred | Accepted, ordered, and handed to the WAL writer. | Nothing | +| Strict | Durable. Recovery will rebuild it. | One storage sync | + +Strict mode gains batched submission. Deferred mode is new. None is the +existing behavior of a host that does not enable a WAL. + +### Batched submission + +A host submits several envelopes in one call. Echo appends the ordinary intake +transaction of each envelope, in order, and then syncs storage once. Echo +acknowledges the batch only after that sync. + +The bytes on disk are the same transactions that single submission writes +today. Recovery does not change. + +A batch is not atomic across a crash. Each transaction is atomic through its +own commit record, and a torn tail is truncated as it is today. A crash before +the sync can therefore leave a prefix of the batch. Echo acknowledged none of +it, so each surviving submission recovers as accepted and pending. That is the +same posture as a single submission whose process died after its commit and +before its acknowledgement. + +When the final sync fails, every submission in the batch is uncertain. Echo +reconciles each one against the log, as it already does after a failed single +append, and does not report an unreconciled submission as accepted. + +### Deferred mode + +In deferred mode a writer thread inside the Echo process owns the WAL store +and its writer-epoch lease. The runtime hands it records through a bounded +queue and does not touch storage itself. + +An acknowledgement carries the log position of the record. It states that the +record is accepted and not yet durable. + +The writer appends queued records in order and syncs them in groups. After +each sync it publishes the durable position: every record at or before that +position will be rebuilt by recovery. A host may read the durable position, or +wait until it passes a given acknowledgement. + +A crash in deferred mode loses the records after the durable position. +Recovery rebuilds a valid prefix of what the runtime accepted. It never +rebuilds a gap. + +## Laws + +1. **An acknowledgement states its posture.** Strict and deferred + acknowledgements are different values. A caller cannot mistake accepted for + durable. +2. **Durability reports are observations.** The durable position, queue depth + and writer timing never influence which intents are admitted, how a tick + is scheduled, or what a tick computes. Two runs that accept the same + intents in the same order produce the same history in every mode. +3. **Order is fixed at acceptance.** The log order is the order in which the + runtime accepted records. The writer never reorders, merges or drops. +4. **Nothing is dropped silently.** When the queue is full, the caller waits + or receives an explicit obstruction. When the writer fails, the runtime + stops accepting work that would need the WAL and reports an obstruction. + It does not continue in a weaker mode. +5. **The log is an ordinary Echo WAL.** Every mode that writes a WAL writes + the same records. One recovery path reads all of them. +6. **One writer.** In every mode exactly one writer holds the epoch lease. + Deferred mode moves that writer to a thread. It does not add a second one. +7. **A mode is declared.** The mode is part of host configuration and is + visible in evidence. A receipt produced without a WAL, or before its + record was durable, does not claim strict durability. + +## Alternatives not chosen + +- **An external WAL process on a port.** It gives the same speed as deferred + mode and adds a wire protocol, a second process to deploy and a second + holder of storage authority. A process crash would also lose records that + an in-process writer had already handed to the operating system. +- **One transaction that carries many acceptances.** It makes a batch atomic + across a crash. It needs a new transaction shape and new recovery rules. + Batched submission with ordinary transactions gives the same sync count + without a format change. This remains an open decision if a host needs + all-or-nothing batches. + +## Open decisions + +- Whether a strict frame should cover its submissions and its tick with one + sync instead of two. +- The policy when the deferred queue is full: block the caller, or refuse + with an obstruction. +- Whether a host may change mode while a runtime is live, and what evidence + the change leaves. + +## Current implementation + +The runtime does not implement this design. + +- A host without a WAL runs and recovers nothing. This is mode None. +- `submit_intent_with_runtime_wal_ack` writes one submission-intake + transaction and returns after that transaction's commit record is synced. + There is no call that accepts several envelopes. +- A scheduler pass writes one tick transaction and syncs it before outcomes + become visible. The filesystem store refuses a pass that would need more + than one tick transaction. +- A strict filesystem commit forces one sync: the commit record. +- `WalDurabilityMode` names `StrictFilesystem`, `StrictObjectStore`, + `Buffered`, `ReadOnlyRecovery` and `Disabled`. The trusted host reports + `Buffered` for its in-memory store and `StrictFilesystem` for its filesystem + store. No mode defers a sync, and no acknowledgement carries a durability + posture or a log position. +- All WAL writes happen on the caller's thread. + +Implementation state is owned by +[issue #787](https://github.com/flyingrobots/echo/issues/787), not by this +page. diff --git a/docs/topics/WAL.md b/docs/topics/WAL.md index d019498d..22dc3b13 100644 --- a/docs/topics/WAL.md +++ b/docs/topics/WAL.md @@ -15,6 +15,10 @@ The short rule is: Echo may only claim what its WAL can recover. ``` +A proposed, not yet implemented design for choosing when Echo waits for +storage is in +[WAL durability modes and batched submission](../architecture/wal-durability-modes.md). + ## What We Found The current runtime WAL evidence says twelve concrete things. From fcb881eefe0cc5d63c59bed4676d0cb6bf69f6d1 Mon Sep 17 00:00:00 2001 From: James Ross Date: Mon, 12 Oct 2026 20:20:24 -0700 Subject: [PATCH 2/7] docs: tighten the proposed WAL durability design after review A failed sync acknowledges nothing: a readable record is not a durable one. Deferred acknowledgements are bound to writer epoch and transaction identity, because a lost record's log position can be reused. Records after the durable position may be lost, not are lost. A full queue makes the caller wait and never refuses, so writer timing cannot decide admission. The one-writer law applies to modes that write a WAL. Only scheduler passes that produce receipt correlations write and sync a tick transaction. Record that the current single-submission path treats a readable record as accepted after a failed append or sync. Co-Authored-By: Claude Opus 5.5 --- docs/architecture/wal-durability-modes.md | 63 +++++++++++++++-------- 1 file changed, 42 insertions(+), 21 deletions(-) diff --git a/docs/architecture/wal-durability-modes.md b/docs/architecture/wal-durability-modes.md index 36964635..90203a6d 100644 --- a/docs/architecture/wal-durability-modes.md +++ b/docs/architecture/wal-durability-modes.md @@ -17,8 +17,9 @@ A host that runs Echo at a fixed frame rate has a fixed time for each frame. At 60 frames per second that time is 16.67 ms. A strict WAL acknowledgement waits for a storage sync. Today each submission -is one transaction with one sync, and each scheduler pass is another. A frame -with many submissions therefore waits for many syncs, one after another. The +is one transaction with one sync, and each scheduler pass that produces +WAL-backed outcomes is another. A frame with many submissions therefore waits +for many syncs, one after another. The count of syncs, not the work of the tick, decides whether the frame fits. The rule of the WAL topic stays in force: Echo may only claim what its WAL can @@ -54,9 +55,12 @@ it, so each surviving submission recovers as accepted and pending. That is the same posture as a single submission whose process died after its commit and before its acknowledgement. -When the final sync fails, every submission in the batch is uncertain. Echo -reconciles each one against the log, as it already does after a failed single -append, and does not report an unreconciled submission as accepted. +When the final sync fails, Echo acknowledges nothing. The records may be +readable in the log, because the operating system can still hold them, but a +readable record is not a durable one. Echo reports a submission of that batch +as durable only after a later sync of the same segment succeeds, or reports +an explicit indeterminate result. It does not turn a readable record into a +strict acknowledgement. ### Deferred mode @@ -64,17 +68,27 @@ In deferred mode a writer thread inside the Echo process owns the WAL store and its writer-epoch lease. The runtime hands it records through a bounded queue and does not touch storage itself. -An acknowledgement carries the log position of the record. It states that the +An acknowledgement names the record by identity: its writer epoch and its +transaction. It also carries the record's log position. It states that the record is accepted and not yet durable. +Identity matters because a position can be reused. If a crash loses an +accepted record, recovery resumes from the last recovered position and a +different record can take the lost one's place. A wait for durability is +satisfied only by the record with the same identity, and recovery reports a +lost acknowledgement as lost. + The writer appends queued records in order and syncs them in groups. After each sync it publishes the durable position: every record at or before that position will be rebuilt by recovery. A host may read the durable position, or wait until it passes a given acknowledgement. -A crash in deferred mode loses the records after the durable position. -Recovery rebuilds a valid prefix of what the runtime accepted. It never -rebuilds a gap. +A crash in deferred mode may lose records after the durable position. It +does not always lose them: the operating system may have written a further +prefix that Echo never promised. Recovery rebuilds a valid prefix of what the +runtime accepted, at least up to the durable position and possibly beyond it. +It never rebuilds a gap. A caller must therefore treat a record after the +durable position as indeterminate until recovery reports it. ## Laws @@ -83,18 +97,21 @@ rebuilds a gap. durable. 2. **Durability reports are observations.** The durable position, queue depth and writer timing never influence which intents are admitted, how a tick - is scheduled, or what a tick computes. Two runs that accept the same + is scheduled, or what a tick computes. Two runs that are offered the same intents in the same order produce the same history in every mode. 3. **Order is fixed at acceptance.** The log order is the order in which the runtime accepted records. The writer never reorders, merges or drops. -4. **Nothing is dropped silently.** When the queue is full, the caller waits - or receives an explicit obstruction. When the writer fails, the runtime - stops accepting work that would need the WAL and reports an obstruction. - It does not continue in a weaker mode. +4. **Nothing is dropped silently, and pressure never refuses.** When the + queue is full, the caller waits. A full queue never refuses an intent, + because that would let writer timing decide what is admitted and would + break law 2. When the writer fails, the runtime stops accepting work that + would need the WAL and reports an explicit obstruction that is recorded as + a fault. It does not continue in a weaker mode. 5. **The log is an ordinary Echo WAL.** Every mode that writes a WAL writes the same records. One recovery path reads all of them. -6. **One writer.** In every mode exactly one writer holds the epoch lease. - Deferred mode moves that writer to a thread. It does not add a second one. +6. **One writer.** In every mode that writes a WAL, exactly one writer holds + the epoch lease. Deferred mode moves that writer to a thread. It does not + add a second one. Mode None has no WAL and no lease. 7. **A mode is declared.** The mode is part of host configuration and is visible in evidence. A receipt produced without a WAL, or before its record was durable, does not claim strict durability. @@ -115,8 +132,8 @@ rebuilds a gap. - Whether a strict frame should cover its submissions and its tick with one sync instead of two. -- The policy when the deferred queue is full: block the caller, or refuse - with an obstruction. +- How long a caller may wait on a full deferred queue before the host treats + the writer as failed. - Whether a host may change mode while a runtime is live, and what evidence the change leaves. @@ -128,9 +145,13 @@ The runtime does not implement this design. - `submit_intent_with_runtime_wal_ack` writes one submission-intake transaction and returns after that transaction's commit record is synced. There is no call that accepts several envelopes. -- A scheduler pass writes one tick transaction and syncs it before outcomes - become visible. The filesystem store refuses a pass that would need more - than one tick transaction. +- A scheduler pass that produces receipt correlations writes one tick + transaction and syncs it before outcomes become visible. A pass that + produces none writes nothing. The filesystem store refuses a pass that would + need more than one tick transaction. +- After a failed append or sync of a single submission, the host re-reads the + log and treats a readable acceptance record as accepted. That is weaker than + the rule this page proposes for a failed sync. - A strict filesystem commit forces one sync: the commit record. - `WalDurabilityMode` names `StrictFilesystem`, `StrictObjectStore`, `Buffered`, `ReadOnlyRecovery` and `Disabled`. The trusted host reports From 10360ba42063f188cd13d4edaff5e48a27012475 Mon Sep 17 00:00:00 2001 From: James Ross Date: Mon, 12 Oct 2026 22:37:07 -0700 Subject: [PATCH 3/7] docs: state that batched submission is implemented Batched submission landed in #791. Correct the status line and the current implementation section, and replace the proposal that a later sync may make a failed batch durable: a later sync cannot vouch for records an earlier sync failed to write, so the host takes the batch back and must reopen the WAL. Co-Authored-By: Claude Opus 5.5 --- docs/architecture/wal-durability-modes.md | 50 ++++++++++++++--------- 1 file changed, 31 insertions(+), 19 deletions(-) diff --git a/docs/architecture/wal-durability-modes.md b/docs/architecture/wal-durability-modes.md index 90203a6d..5deb7e0b 100644 --- a/docs/architecture/wal-durability-modes.md +++ b/docs/architecture/wal-durability-modes.md @@ -3,7 +3,8 @@ # WAL Durability Modes and Batched Submission -- **Status:** Proposed design. **Not implemented.** The section +- **Status:** Proposed design. Batched submission is implemented; deferred + mode and acknowledgement postures are **not implemented**. The section [Current implementation](#current-implementation) states what the runtime does today. - **Proposal date:** 2026-10-12 @@ -16,11 +17,11 @@ A host that runs Echo at a fixed frame rate has a fixed time for each frame. At 60 frames per second that time is 16.67 ms. -A strict WAL acknowledgement waits for a storage sync. Today each submission -is one transaction with one sync, and each scheduler pass that produces -WAL-backed outcomes is another. A frame with many submissions therefore waits -for many syncs, one after another. The -count of syncs, not the work of the tick, decides whether the frame fits. +A strict WAL acknowledgement waits for a storage sync. A submission made on its +own is one transaction with one sync, and each scheduler pass that produces +WAL-backed outcomes is another. A frame that submits many intents one at a time +therefore waits for many syncs, one after another. The count of syncs, not the +work of the tick, decides whether the frame fits. The rule of the WAL topic stays in force: Echo may only claim what its WAL can recover. This page changes when Echo waits and what an acknowledgement says. @@ -36,7 +37,7 @@ A host chooses one of three durability modes for a runtime. | Deferred | Accepted, ordered, and handed to the WAL writer. | Nothing | | Strict | Durable. Recovery will rebuild it. | One storage sync | -Strict mode gains batched submission. Deferred mode is new. None is the +Strict mode has batched submission. Deferred mode is new. None is the existing behavior of a host that does not enable a WAL. ### Batched submission @@ -57,10 +58,11 @@ before its acknowledgement. When the final sync fails, Echo acknowledges nothing. The records may be readable in the log, because the operating system can still hold them, but a -readable record is not a durable one. Echo reports a submission of that batch -as durable only after a later sync of the same segment succeeds, or reports -an explicit indeterminate result. It does not turn a readable record into a -strict acknowledgement. +readable record is not a durable one. A later sync cannot vouch for records +that an earlier sync failed to write, so the running host does not retry: it +takes the batch back, refuses further WAL work, and must reopen the WAL. A +host that opens a log syncs it before reading it as recovered history. It does +not turn a readable record into a strict acknowledgement. ### Deferred mode @@ -139,26 +141,36 @@ durable position as indeterminate until recovery reports it. ## Current implementation -The runtime does not implement this design. +The runtime implements batched submission. It does not implement deferred +mode or acknowledgement postures. - A host without a WAL runs and recovers nothing. This is mode None. - `submit_intent_with_runtime_wal_ack` writes one submission-intake transaction and returns after that transaction's commit record is synced. - There is no call that accepts several envelopes. +- `submit_intents_with_runtime_wal_ack` accepts several envelopes, appends one + ordinary intake transaction for each, and returns after one sync covers all + of them. Until that sync succeeds no acceptance of the batch counts as + durable or can be scheduled. [WAL](../topics/WAL.md) states its failure + rules. +- After a storage failure in a batch the adapter refuses further WAL work with + `StorageFailed` until the host reopens the WAL. It does not reconcile itself + from the log, answer recovery reads, or offer its cached causal frontier or + evidence catalog. +- A host that opens an existing log, and a writer that takes it over, sync its + segment files before reading them as recovered history. +- After a failed append or sync of a *single* submission, the host still + re-reads the log and treats a readable acceptance record as accepted. That is + weaker than the batch rule and than the rule this page states. - A scheduler pass that produces receipt correlations writes one tick transaction and syncs it before outcomes become visible. A pass that produces none writes nothing. The filesystem store refuses a pass that would need more than one tick transaction. -- After a failed append or sync of a single submission, the host re-reads the - log and treats a readable acceptance record as accepted. That is weaker than - the rule this page proposes for a failed sync. - A strict filesystem commit forces one sync: the commit record. - `WalDurabilityMode` names `StrictFilesystem`, `StrictObjectStore`, `Buffered`, `ReadOnlyRecovery` and `Disabled`. The trusted host reports `Buffered` for its in-memory store and `StrictFilesystem` for its filesystem - store. No mode defers a sync, and no acknowledgement carries a durability - posture or a log position. -- All WAL writes happen on the caller's thread. + store. No acknowledgement carries a durability posture or a log position. +- All WAL writes happen on the caller's thread. No writer thread exists. Implementation state is owned by [issue #787](https://github.com/flyingrobots/echo/issues/787), not by this From c8732013efe19dbc7d5672d4910cfb234906a0bc Mon Sep 17 00:00:00 2001 From: James Ross Date: Tue, 13 Oct 2026 09:21:03 -0700 Subject: [PATCH 4/7] Fix: state queue waits, mid-batch failure, batch limits and fault posture in the WAL design Address five review threads on the proposed WAL durability design. The mode table promised that a deferred caller waits for nothing, while a later law makes it wait when the queue is full. State the wait and how it ends. Writer failure was called both an obstruction and a fault; they are separate typed values with different recovery. A failure in the middle of a batch had no stated result; state both cases as batched submission now implements them. A batch had no size bound; propose a host-set limit checked before the first append, and state that none exists today. The rejected external-writer alternative conflated a host crash with a writer crash. Refs #787. Co-Authored-By: Claude Opus 5.5 --- docs/architecture/wal-durability-modes.md | 54 ++++++++++++++++++----- 1 file changed, 43 insertions(+), 11 deletions(-) diff --git a/docs/architecture/wal-durability-modes.md b/docs/architecture/wal-durability-modes.md index 5deb7e0b..d05d2104 100644 --- a/docs/architecture/wal-durability-modes.md +++ b/docs/architecture/wal-durability-modes.md @@ -31,11 +31,17 @@ It does not change what Echo may claim. A host chooses one of three durability modes for a runtime. -| Mode | An acknowledgement means | The caller waits for | -| -------- | ---------------------------------------------------------- | -------------------- | -| None | Accepted in memory. Nothing is recoverable. | Nothing | -| Deferred | Accepted, ordered, and handed to the WAL writer. | Nothing | -| Strict | Durable. Recovery will rebuild it. | One storage sync | +| Mode | An acknowledgement means | The caller waits for | +| -------- | ------------------------------------------------ | ----------------------------------- | +| None | Accepted in memory. Nothing is recoverable. | Nothing | +| Deferred | Accepted, ordered, and handed to the WAL writer. | Room in the writer's queue, if full | +| Strict | Durable. Recovery will rebuild it. | One storage sync | + +A deferred caller normally waits for nothing. When the writer's queue is full +it waits for room, and that wait ends in one of two ways: room appears, or +the host declares the writer failed and the caller receives that failure. A +deferred acknowledgement is therefore not a promise of a bounded wait until +the limit in [Open decisions](#open-decisions) is set. Strict mode has batched submission. Deferred mode is new. None is the existing behavior of a host that does not enable a WAL. @@ -56,6 +62,20 @@ it, so each surviving submission recovers as accepted and pending. That is the same posture as a single submission whose process died after its commit and before its acknowledgement. +A batch has a size the host sets: a limit on envelopes and on encoded bytes. +Echo checks the whole batch against that limit before it appends anything. A +batch over the limit is refused as a whole with a typed over-budget result. + +An envelope can fail in the middle of a batch. The result depends on where: + +- **Before anything is written for it.** The runtime or the WAL refuses the + envelope. The envelopes before it are synced and stay accepted. The call + returns the refusal and no acknowledgement. A caller that submits an + earlier envelope again receives its existing submission. +- **While it is written, or at the sync.** Storage failed. Echo takes back + every submission the call accepted, acknowledges nothing, and refuses + further WAL work until the WAL is reopened. + When the final sync fails, Echo acknowledges nothing. The records may be readable in the log, because the operating system can still hold them, but a readable record is not a durable one. A later sync cannot vouch for records @@ -106,9 +126,14 @@ durable position as indeterminate until recovery reports it. 4. **Nothing is dropped silently, and pressure never refuses.** When the queue is full, the caller waits. A full queue never refuses an intent, because that would let writer timing decide what is admitted and would - break law 2. When the writer fails, the runtime stops accepting work that - would need the WAL and reports an explicit obstruction that is recorded as - a fault. It does not continue in a weaker mode. + break law 2. When the writer fails, that is an internal fault: Echo did + not uphold its own durability. The runtime records the fault and stops + accepting work that would need the WAL. A caller whose work is refused for + that reason receives an obstruction that names the fault. The two stay + separate typed values, as [Obstructions](../topics/Obstructions.md) + requires: the fault needs trusted recovery, and the obstruction tells the + caller that support is unavailable. The runtime does not continue in a + weaker mode. 5. **The log is an ordinary Echo WAL.** Every mode that writes a WAL writes the same records. One recovery path reads all of them. 6. **One writer.** In every mode that writes a WAL, exactly one writer holds @@ -122,8 +147,13 @@ durable position as indeterminate until recovery reports it. - **An external WAL process on a port.** It gives the same speed as deferred mode and adds a wire protocol, a second process to deploy and a second - holder of storage authority. A process crash would also lose records that - an in-process writer had already handed to the operating system. + holder of storage authority. Its failures differ from an in-process + writer's. If the Echo host dies after handing a record over, the external + writer can still sync it, where an in-process writer dies with the host. If + the external writer dies, records it had accepted and not written are lost + while the host runs on. Each case needs its own acknowledgement and + recovery rule, and an acknowledgement would have to say which process it + binds. That is a larger contract than this design needs. - **One transaction that carries many acceptances.** It makes a batch atomic across a crash. It needs a new transaction shape and new recovery rules. Batched submission with ordinary transactions gives the same sync count @@ -152,13 +182,15 @@ mode or acknowledgement postures. of them. Until that sync succeeds no acceptance of the batch counts as durable or can be scheduled. [WAL](../topics/WAL.md) states its failure rules. +- A batch has no size limit. Nothing checks envelope count or bytes before + the first append. - After a storage failure in a batch the adapter refuses further WAL work with `StorageFailed` until the host reopens the WAL. It does not reconcile itself from the log, answer recovery reads, or offer its cached causal frontier or evidence catalog. - A host that opens an existing log, and a writer that takes it over, sync its segment files before reading them as recovered history. -- After a failed append or sync of a *single* submission, the host still +- After a failed append or sync of a _single_ submission, the host still re-reads the log and treats a readable acceptance record as accepted. That is weaker than the batch rule and than the rule this page states. - A scheduler pass that produces receipt correlations writes one tick From 25b463123a1a15c256d739d213a44099a851b92c Mon Sep 17 00:00:00 2001 From: James Ross Date: Tue, 13 Oct 2026 14:45:14 -0700 Subject: [PATCH 5/7] Docs: say what each store waits for and qualify the sync guarantee The in-memory batch call still waits for commitment in memory; it has no storage sync. A filesystem acknowledgement is as strong as the sync guarantee of the operating system and device under the store. Co-Authored-By: Claude Opus 5.5 --- docs/architecture/batched-strict-submission.md | 17 ++++++++++------- 1 file changed, 10 insertions(+), 7 deletions(-) diff --git a/docs/architecture/batched-strict-submission.md b/docs/architecture/batched-strict-submission.md index 9f87e326..21f944a8 100644 --- a/docs/architecture/batched-strict-submission.md +++ b/docs/architecture/batched-strict-submission.md @@ -42,13 +42,16 @@ Recovery reads a batch exactly as it reads single submissions. The batch call is the same for every store. Its durability is not. -| Store | Reported mode | What the batch call waits for | What survives a crash | -| ---------- | ------------------ | ---------------------------------------- | ------------------------- | -| Filesystem | `StrictFilesystem` | One storage sync | Every acknowledged record | -| In-memory | `Buffered` | Nothing; no sync exists | Nothing | -| No WAL | not applicable | The call returns `RuntimeWalUnavailable` | Nothing | - -An acknowledgement from the filesystem store is a durability claim. An +| Store | Reported mode | What the batch call waits for | What survives a crash | +| ---------- | ------------------ | -------------------------------------------- | ---------------------------------------------------- | +| Filesystem | `StrictFilesystem` | Commitment in memory, then one storage sync | Every acknowledged record, as far as that sync holds | +| In-memory | `Buffered` | Commitment in memory; no storage sync exists | Nothing | +| No WAL | not applicable | The call returns `RuntimeWalUnavailable` | Nothing | + +An acknowledgement from the filesystem store is a durability claim. It is as +strong as the sync guarantee of the operating system and the device under the +store: Echo asks for the sync and waits for it, and cannot promise more than +that sync delivers. An acknowledgement from the in-memory store is not: it says the runtime accepted and ordered the records, and nothing more. `Buffered` is not release durability. A host must read the reported mode before it tells anyone that an From 32a9d95fd90d1159b5ba6edbe6dcd1855364bbe4 Mon Sep 17 00:00:00 2001 From: James Ross Date: Tue, 13 Oct 2026 14:51:03 -0700 Subject: [PATCH 6/7] Docs: one owner for the batch contract, zero-write batches, witnesses - The WAL topic owns the crash, refusal, storage-failure and reopen contract. The architecture page summarises it and links there. - A duplicate envelope appends nothing; an empty batch or a batch of duplicates writes nothing and forces no sync. Both pages say so. - Both pages name the batch API source and its six tests. Co-Authored-By: Claude Opus 5.5 --- .../architecture/batched-strict-submission.md | 65 +++++++++---------- docs/topics/WAL.md | 9 +++ 2 files changed, 41 insertions(+), 33 deletions(-) diff --git a/docs/architecture/batched-strict-submission.md b/docs/architecture/batched-strict-submission.md index 21f944a8..372a658a 100644 --- a/docs/architecture/batched-strict-submission.md +++ b/docs/architecture/batched-strict-submission.md @@ -27,10 +27,15 @@ recover. ## What a host can do A host submits several envelopes in one call, -`submit_intents_with_runtime_wal_ack`. Echo appends the ordinary intake -transaction of each envelope, in order, and then asks the store to make them -durable once. Echo acknowledges the batch only after that step succeeds. Until -then no acceptance of the batch counts as durable and none can be scheduled. +`submit_intents_with_runtime_wal_ack`. Echo appends one ordinary intake +transaction for each newly accepted envelope, in order, and then asks the +store to make them durable once. Echo acknowledges the batch only after that +step succeeds. Until then no acceptance of the batch counts as durable and +none can be scheduled. + +An envelope that repeats a submission Echo already holds is a duplicate. It +appends nothing and its handle names the existing submission. An empty batch, +and a batch of duplicates only, therefore writes nothing and forces no sync. `submit_intent_with_runtime_wal_ack` still submits one envelope with one transaction and one sync. @@ -60,37 +65,18 @@ acknowledged record will survive a crash. No acknowledgement carries its own durability posture or a log position today. The mode is a property of the configured store, not of the returned value. -## A batch is not atomic across a crash - -Each transaction is atomic through its own commit record, and a torn tail is -truncated as it is for single submissions. A crash before the sync can -therefore leave a prefix of the batch. Echo acknowledged none of it, so each -surviving submission recovers as accepted and pending. That is the same -posture as a single submission whose process died after its commit and before -its acknowledgement. - -## Failure inside a batch +## Crash, refusal and storage failure -An envelope can fail in the middle of a batch. The result depends on where: +[WAL](../topics/WAL.md) owns the complete contract for a batch: what a crash +can leave, what happens when an intent is refused in the middle, what a +storage failure takes back, and what a host must do when it opens an existing +log. This page does not restate it. In short: -- **Before anything is written for it.** The runtime or the WAL refuses the - envelope. The envelopes before it are synced and stay accepted. The call - returns the refusal and no acknowledgement. A caller that submits an - earlier envelope again receives its existing submission. -- **While it is written, or at the sync.** Storage failed. Echo takes back - every submission the call accepted, acknowledges nothing, and refuses - further WAL work with `StorageFailed` until the host reopens the WAL. - -When the final sync fails, the records may still be readable in the log, -because the operating system can hold them. A readable record is not a durable -one, and a later sync cannot vouch for records that an earlier sync failed to -write. The running host therefore does not retry and does not reconcile itself -from the log. After the failure it does not answer recovery reads or offer its -cached causal frontier or evidence catalog. - -A host that opens an existing log, and a writer that takes one over, sync its -segment files before reading them as recovered history. That sync does not -turn a readable record into a strict acknowledgement. +- A batch is not atomic across a crash. A prefix can survive, none of it + acknowledged. +- A refusal before anything is written keeps the earlier intents. +- A storage failure takes back everything the call accepted and stops WAL + work until the host reopens the WAL. ## Limits of the current implementation @@ -114,3 +100,16 @@ Work on these limits is tracked in atomic across a crash. It needs a new transaction shape and new recovery rules. Batching ordinary transactions gives the same sync count without a format change. + +## Evidence + +- API: `submit_intents_with_runtime_wal_ack` in + [`crates/warp-core/src/trusted_runtime_host.rs`](../../crates/warp-core/src/trusted_runtime_host.rs). +- Witnesses, in + [`crates/warp-core/tests/trusted_runtime_host_loop_tests.rs`](../../crates/warp-core/tests/trusted_runtime_host_loop_tests.rs): + - `batched_submission_makes_every_acceptance_durable_with_one_sync` + - `batched_submission_handles_duplicates_and_an_empty_batch` + - `batched_submission_refused_before_storage_keeps_earlier_intents_and_the_wal` + - `batched_submission_refused_before_storage_keeps_an_in_memory_wal_usable` + - `batched_submission_append_failure_fails_closed_until_the_wal_is_reopened` + - `batched_submission_sync_failure_rolls_back_intake_and_fails_closed` diff --git a/docs/topics/WAL.md b/docs/topics/WAL.md index 51ca436c..914869f1 100644 --- a/docs/topics/WAL.md +++ b/docs/topics/WAL.md @@ -37,6 +37,10 @@ succeeds, no acceptance of the batch counts as durable and none can be scheduled. A batch is not atomic across a crash: a prefix can survive, none of it acknowledged, and each survivor recovers as `AcceptedPending`. +An intent that repeats a submission Echo already holds is a duplicate: it +writes no transaction and its handle names the existing submission. An empty +batch, and a batch of duplicates only, writes nothing and forces no sync. + If the runtime or the WAL refuses an intent of a batch before anything is written for it, the intents before it are still synced and stay accepted; the call returns the refusal and no handles. @@ -413,6 +417,11 @@ The runtime ACK and recovery witnesses live in The core test names to read first are: +- `batched_submission_makes_every_acceptance_durable_with_one_sync` +- `batched_submission_handles_duplicates_and_an_empty_batch` +- `batched_submission_refused_before_storage_keeps_earlier_intents_and_the_wal` +- `batched_submission_append_failure_fails_closed_until_the_wal_is_reopened` +- `batched_submission_sync_failure_rolls_back_intake_and_fails_closed` - `runtime_wal_ack_submit_commits_acceptance_before_returning_handle` - `runtime_wal_ack_path_requires_configured_runtime_wal` - `runtime_wal_ack_failure_rolls_back_intake_mutation` From 0efcc19dba2e2a64c202628222816c43bcec536b Mon Sep 17 00:00:00 2001 From: James Ross Date: Tue, 13 Oct 2026 14:57:38 -0700 Subject: [PATCH 7/7] Docs: scope sync and failure claims to the store that has them - The WAL topic summary says only the filesystem store syncs storage. - A duplicate single submission writes and syncs nothing. - The weaker single-submission failure rule applies to the filesystem store; the in-memory store takes the submission back. Co-Authored-By: Claude Opus 5.5 --- docs/architecture/batched-strict-submission.md | 14 +++++++++----- docs/topics/WAL.md | 5 +++-- 2 files changed, 12 insertions(+), 7 deletions(-) diff --git a/docs/architecture/batched-strict-submission.md b/docs/architecture/batched-strict-submission.md index 372a658a..4381ebc0 100644 --- a/docs/architecture/batched-strict-submission.md +++ b/docs/architecture/batched-strict-submission.md @@ -37,8 +37,10 @@ An envelope that repeats a submission Echo already holds is a duplicate. It appends nothing and its handle names the existing submission. An empty batch, and a batch of duplicates only, therefore writes nothing and forces no sync. -`submit_intent_with_runtime_wal_ack` still submits one envelope with one -transaction and one sync. +`submit_intent_with_runtime_wal_ack` still submits one envelope. A newly +accepted envelope writes one transaction and, on the filesystem store, waits +for one sync. A duplicate returns the existing handle, writes nothing and +syncs nothing. The bytes in the log are the same transactions that single submission writes. Recovery reads a batch exactly as it reads single submissions. @@ -82,9 +84,11 @@ log. This page does not restate it. In short: - **A batch has no size limit.** Nothing checks envelope count or encoded bytes before the first append. -- **Single submission is weaker after a failure.** After a failed append or - sync of a single submission, the host still re-reads the log and treats a - readable acceptance record as accepted. The batch path does not. +- **Single submission on the filesystem store is weaker after a failure.** + After a failed append or sync of a single submission, a host on the + filesystem store re-reads the log and treats a readable acceptance record + as accepted. The batch path does not. A host on the in-memory store has no + log to re-read: it takes the submission back. - **A tick is a separate sync.** A scheduler pass that produces receipt correlations writes one tick transaction and syncs it before outcomes become visible. A pass that produces none writes nothing. A strict frame diff --git a/docs/topics/WAL.md b/docs/topics/WAL.md index 914869f1..11da8dc8 100644 --- a/docs/topics/WAL.md +++ b/docs/topics/WAL.md @@ -16,8 +16,9 @@ Echo may only claim what its WAL can recover. ``` [Batched strict submission](../architecture/batched-strict-submission.md) -states how a host submits several intents behind one storage sync, and what -an acknowledgement means for each store. +states how a host submits several intents behind one commit step, and what +an acknowledgement means for each store. Only the filesystem store syncs +storage in that step; the in-memory store has no storage to sync. ## What We Found