diff --git a/docs/README.md b/docs/README.md index d33d8555..0bfd2fd0 100644 --- a/docs/README.md +++ b/docs/README.md @@ -20,6 +20,7 @@ causal history. Git history is the archive; GitHub owns live work and status. - [There Is No Graph](architecture/there-is-no-graph.md) - [Sliced readings and copy-on-write strands](architecture/sliced-readings-and-copy-on-write-strands.md) (accepted design, not implemented) +- [Batched strict submission](architecture/batched-strict-submission.md) - [Participant history as a braid](architecture/participant-history-as-a-braid.md) (accepted design, not implemented) - [Continuum transport](architecture/continuum-transport.md) - [Echo optics adapter notes](architecture/echo-optics-adapter-notes.md) diff --git a/docs/architecture/batched-strict-submission.md b/docs/architecture/batched-strict-submission.md new file mode 100644 index 00000000..4381ebc0 --- /dev/null +++ b/docs/architecture/batched-strict-submission.md @@ -0,0 +1,119 @@ + + + +# Batched Strict Submission + +- **Status:** Implemented. +- **Refines:** [WAL](../topics/WAL.md) +- **Related:** [Fixed timestep invariant](../invariants/FIXED-TIMESTEP.md) + +This page states what the runtime does today. A proposal for a deferred +durability mode is not part of it. That proposal and its open decisions live in +[issue #787](https://github.com/flyingrobots/echo/issues/787). + +## Problem + +A host that runs Echo at a fixed frame rate has a fixed time for each frame. +At 60 frames per second that time is 16.67 ms. + +A strict WAL acknowledgement waits for a storage sync. A submission made on its +own is one transaction with one sync. A frame that submits many intents one at +a time therefore waits for many syncs, one after another. The count of syncs, +not the work of the tick, decides whether the frame fits. + +The rule of the WAL topic stays in force: Echo may only claim what its WAL can +recover. + +## What a host can do + +A host submits several envelopes in one call, +`submit_intents_with_runtime_wal_ack`. Echo appends one ordinary intake +transaction for each newly accepted envelope, in order, and then asks the +store to make them durable once. Echo acknowledges the batch only after that +step succeeds. Until then no acceptance of the batch counts as durable and +none can be scheduled. + +An envelope that repeats a submission Echo already holds is a duplicate. It +appends nothing and its handle names the existing submission. An empty batch, +and a batch of duplicates only, therefore writes nothing and forces no sync. + +`submit_intent_with_runtime_wal_ack` still submits one envelope. A newly +accepted envelope writes one transaction and, on the filesystem store, waits +for one sync. A duplicate returns the existing handle, writes nothing and +syncs nothing. + +The bytes in the log are the same transactions that single submission writes. +Recovery reads a batch exactly as it reads single submissions. + +## What an acknowledgement means depends on the store + +The batch call is the same for every store. Its durability is not. + +| Store | Reported mode | What the batch call waits for | What survives a crash | +| ---------- | ------------------ | -------------------------------------------- | ---------------------------------------------------- | +| Filesystem | `StrictFilesystem` | Commitment in memory, then one storage sync | Every acknowledged record, as far as that sync holds | +| In-memory | `Buffered` | Commitment in memory; no storage sync exists | Nothing | +| No WAL | not applicable | The call returns `RuntimeWalUnavailable` | Nothing | + +An acknowledgement from the filesystem store is a durability claim. It is as +strong as the sync guarantee of the operating system and the device under the +store: Echo asks for the sync and waits for it, and cannot promise more than +that sync delivers. An +acknowledgement from the in-memory store is not: it says the runtime accepted +and ordered the records, and nothing more. `Buffered` is not release +durability. A host must read the reported mode before it tells anyone that an +acknowledged record will survive a crash. + +No acknowledgement carries its own durability posture or a log position today. +The mode is a property of the configured store, not of the returned value. + +## Crash, refusal and storage failure + +[WAL](../topics/WAL.md) owns the complete contract for a batch: what a crash +can leave, what happens when an intent is refused in the middle, what a +storage failure takes back, and what a host must do when it opens an existing +log. This page does not restate it. In short: + +- A batch is not atomic across a crash. A prefix can survive, none of it + acknowledged. +- A refusal before anything is written keeps the earlier intents. +- A storage failure takes back everything the call accepted and stops WAL + work until the host reopens the WAL. + +## Limits of the current implementation + +- **A batch has no size limit.** Nothing checks envelope count or encoded + bytes before the first append. +- **Single submission on the filesystem store is weaker after a failure.** + After a failed append or sync of a single submission, a host on the + filesystem store re-reads the log and treats a readable acceptance record + as accepted. The batch path does not. A host on the in-memory store has no + log to re-read: it takes the submission back. +- **A tick is a separate sync.** A scheduler pass that produces receipt + correlations writes one tick transaction and syncs it before outcomes + become visible. A pass that produces none writes nothing. A strict frame + that submits a batch and then ticks therefore waits for two syncs. +- **All WAL writes happen on the caller's thread.** No writer thread exists. + +Work on these limits is tracked in +[issue #787](https://github.com/flyingrobots/echo/issues/787). + +## Alternative not chosen + +**One transaction that carries many acceptances.** It would make a batch +atomic across a crash. It needs a new transaction shape and new recovery +rules. Batching ordinary transactions gives the same sync count without a +format change. + +## Evidence + +- API: `submit_intents_with_runtime_wal_ack` in + [`crates/warp-core/src/trusted_runtime_host.rs`](../../crates/warp-core/src/trusted_runtime_host.rs). +- Witnesses, in + [`crates/warp-core/tests/trusted_runtime_host_loop_tests.rs`](../../crates/warp-core/tests/trusted_runtime_host_loop_tests.rs): + - `batched_submission_makes_every_acceptance_durable_with_one_sync` + - `batched_submission_handles_duplicates_and_an_empty_batch` + - `batched_submission_refused_before_storage_keeps_earlier_intents_and_the_wal` + - `batched_submission_refused_before_storage_keeps_an_in_memory_wal_usable` + - `batched_submission_append_failure_fails_closed_until_the_wal_is_reopened` + - `batched_submission_sync_failure_rolls_back_intake_and_fails_closed` diff --git a/docs/topics/WAL.md b/docs/topics/WAL.md index 3a5d4fcf..11da8dc8 100644 --- a/docs/topics/WAL.md +++ b/docs/topics/WAL.md @@ -15,6 +15,11 @@ The short rule is: Echo may only claim what its WAL can recover. ``` +[Batched strict submission](../architecture/batched-strict-submission.md) +states how a host submits several intents behind one commit step, and what +an acknowledgement means for each store. Only the filesystem store syncs +storage in that step; the in-memory store has no storage to sync. + ## What We Found The current runtime WAL evidence says twelve concrete things. @@ -33,6 +38,10 @@ succeeds, no acceptance of the batch counts as durable and none can be scheduled. A batch is not atomic across a crash: a prefix can survive, none of it acknowledged, and each survivor recovers as `AcceptedPending`. +An intent that repeats a submission Echo already holds is a duplicate: it +writes no transaction and its handle names the existing submission. An empty +batch, and a batch of duplicates only, writes nothing and forces no sync. + If the runtime or the WAL refuses an intent of a batch before anything is written for it, the intents before it are still synced and stay accepted; the call returns the refusal and no handles. @@ -409,6 +418,11 @@ The runtime ACK and recovery witnesses live in The core test names to read first are: +- `batched_submission_makes_every_acceptance_durable_with_one_sync` +- `batched_submission_handles_duplicates_and_an_empty_batch` +- `batched_submission_refused_before_storage_keeps_earlier_intents_and_the_wal` +- `batched_submission_append_failure_fails_closed_until_the_wal_is_reopened` +- `batched_submission_sync_failure_rolls_back_intake_and_fails_closed` - `runtime_wal_ack_submit_commits_acceptance_before_returning_handle` - `runtime_wal_ack_path_requires_configured_runtime_wal` - `runtime_wal_ack_failure_rolls_back_intake_mutation`