Repository navigation
Propose WAL durability modes and batched submission - #788
flyingrobots wants to merge 4 commits into
Conversation
Record a proposed, not yet implemented design for when Echo waits for storage: three host-chosen modes (none, deferred, strict), batched strict submission that appends ordinary intake transactions and syncs once, and a deferred mode with an in-process writer thread, a bounded queue and an observable durable position. State the laws that keep it honest: acknowledgements state their posture, durability reports never influence admission or ticks, nothing is dropped silently, and every mode writes an ordinary Echo WAL. Record what the runtime does today and name issue #787 as the owner of implementation state. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Codex Review SummaryThis comment shows the latest Codex review activity on this pull request.
ℹ️ About Codex in GitHubYour team has set up Codex to review pull requests in this repo. Reviews are triggered when you
Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings. |
|
Important Draft PR not reviewedDraft PRs are not automatically reviewed by default.
To automatically review draft PRs, update your CodeRabbit configuration: reviews:
auto_review:
drafts: true
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
![chatgpt-codex-connector[bot]](https://avatars.githubusercontent.com/in/1144995?s=60&v=4){kind=small}
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 6fbb7c5ac6
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
| - **Status:** Proposed design. **Not implemented.** The section | ||
| [Current implementation](#current-implementation) states what the runtime | ||
| does today. |
There was a problem hiding this comment.
Because this page explicitly identifies itself as a proposed design and still contains open decisions, checking it into docs/architecture/ and routing it through the Current Architecture index makes undecided behavior appear to be current architectural truth. Keep the proposal in issue #787 until it is accepted, then record the resulting durable decision in its canonical owner.
AGENTS.md reference: AGENTS.md:L69-L75
Useful? React with 👍 / 👎.
There was a problem hiding this comment.
Agreed that a proposal must not read as current architecture. I have converted this PR to a draft and am leaving this thread open. The design is tracked in #787 and will merge only if the repository owner accepts it, at which point the status line changes from proposed to accepted; otherwise the page stays out of docs/architecture/ and the content lives in the issue.
A failed sync acknowledges nothing: a readable record is not a durable one. Deferred acknowledgements are bound to writer epoch and transaction identity, because a lost record's log position can be reused. Records after the durable position may be lost, not are lost. A full queue makes the caller wait and never refuses, so writer timing cannot decide admission. The one-writer law applies to modes that write a WAL. Only scheduler passes that produce receipt correlations write and sync a tick transaction. Record that the current single-submission path treats a readable record as accepted after a failed append or sync. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: fcb881eefe
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
| | Mode | An acknowledgement means | The caller waits for | | ||
| | -------- | ---------------------------------------------------------- | -------------------- | | ||
| | None | Accepted in memory. Nothing is recoverable. | Nothing | | ||
| | Deferred | Accepted, ordered, and handed to the WAL writer. | Nothing | |
There was a problem hiding this comment.
At queue saturation, the Nothing promise is false: law 4 at lines 104–105 requires the submission caller to wait for capacity, and the open decision allows that wait to continue until the host declares writer failure. A deferred client therefore cannot tell whether acknowledgement is nonblocking or may stall a frame—the performance property this design is intended to define. State the queue-capacity wait and its terminal failure result here rather than promising no wait.
AGENTS.md reference: AGENTS.md:L20-L27
Useful? React with 👍 / 👎.
| break law 2. When the writer fails, the runtime stops accepting work that | ||
| would need the WAL and reports an explicit obstruction that is recorded as | ||
| a fault. It does not continue in a weaker mode. |
There was a problem hiding this comment.
On writer failure, this assigns the same condition both obstruction and fault, although docs/topics/RuntimeAuthority.md lines 25–28 says fault/quarantine must not be confused with lawful obstruction and docs/topics/Obstructions.md lines 12–23 gives them different recovery semantics. An implementation following this cannot expose a stable typed posture—retrying unavailable support and quarantining an internally faulted runtime require different handling—so define the public obstruction and internal fault as separate attributable values, or select the one posture that applies.
AGENTS.md reference: AGENTS.md:L65-L75
Useful? React with 👍 / 👎.
| A host submits several envelopes in one call. Echo appends the ordinary intake | ||
| transaction of each envelope, in order, and then syncs storage once. Echo | ||
| acknowledges the batch only after that sync. |
There was a problem hiding this comment.
If transaction N fails validation, construction, or append after earlier transactions were appended, the prefix may later become durable through OS writeback or a subsequent sync even though the batch never reaches this acknowledgement point. The page defines crash and final-sync failure but gives no result posture for this ordinary partial-failure path, so callers cannot know which submissions to retry safely. Require all fallible admission/build work before appending, or return identity-bound accepted/indeterminate results for every prefix member and reconcile them before reporting the batch result.
AGENTS.md reference: AGENTS.md:L30-L34
Useful? React with 👍 / 👎.
| A host submits several envelopes in one call. Echo appends the ordinary intake | ||
| transaction of each envelope, in order, and then syncs storage once. Echo | ||
| acknowledges the batch only after that sync. |
There was a problem hiding this comment.
This new public call has no limit on envelope count, encoded bytes, or admission work, so a caller can supply an arbitrarily large batch and monopolize memory and the runtime thread even though the motivating contract is a fixed frame budget. Define a host-configured batch aperture and a typed over-budget result that is decided before partial append; otherwise reducing sync count does not make the frame-time claim bounded.
AGENTS.md reference: AGENTS.md:L20-L27
Useful? React with 👍 / 👎.
| - **An external WAL process on a port.** It gives the same speed as deferred | ||
| mode and adds a wire protocol, a second process to deploy and a second | ||
| holder of storage authority. A process crash would also lose records that | ||
| an in-process writer had already handed to the operating system. |
There was a problem hiding this comment.
This rationale conflates a host crash with a WAL-daemon crash. If the Echo host dies after an IPC handoff but the external writer remains alive, that writer can still retain and sync the records; the in-process writer necessarily dies with the host. Conversely, daemon failure needs its own acknowledgement and recovery contract. Distinguish which process failed and what the IPC acknowledgement binds rather than claiming the external design has the same loss posture.
Useful? React with 👍 / 👎.
Batched submission landed in #791. Correct the status line and the current implementation section, and replace the proposal that a later sync may make a failed batch durable: a later sync cannot vouch for records an earlier sync failed to write, so the host takes the batch back and must reopen the WAL. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
|
Merged |
1 participant
What changes
Adds a proposed, not yet implemented design page:
docs/architecture/wal-durability-modes.md. Documentation only.Summary
The page has a Current implementation section and names #787 as the owner of implementation state. Linked from
docs/README.mdanddocs/topics/WAL.md.No numbered ADR. No CHANGELOG entry: no shipped behavior changes.
Refs #787.
🤖 Generated with Claude Code