From 252376546c8d22b16cb77e6be2c04a5283016e92 Mon Sep 17 00:00:00 2001 From: Yordis Prieto Date: Wed, 7 Oct 2026 02:40:03 -0400 Subject: [PATCH 1/3] feat(consistency): add minimize_latency and fully_consistent modes Signed-off-by: Yordis Prieto --- docs/explanation/consistency-pattern.md | 217 ++++++++++-------- .../consistency/v1alpha1/consistency.proto | 56 ++++- 2 files changed, 166 insertions(+), 107 deletions(-) diff --git a/docs/explanation/consistency-pattern.md b/docs/explanation/consistency-pattern.md index a54e770..9fd4219 100644 --- a/docs/explanation/consistency-pattern.md +++ b/docs/explanation/consistency-pattern.md @@ -2,61 +2,83 @@ ## Overview -The `trogon.consistency.v1alpha1` package provides Protobuf message definitions for implementing **read-after-write consistency** in eventual consistency systems, particularly CQRS/Event Sourcing architectures. +The `trogon.consistency.v1alpha1` package provides Protobuf message definitions for expressing consistency requirements on queries in event sourced systems. It lets a client state, per query, how the response should relate to the position of the event log: accept whatever a projection currently reflects, wait for a specific position, wait for an exact snapshot, or wait for the head of the log as observed at request time. ## Problem Statement -In CQRS/Event Sourcing systems, there's often a delay (projection lag) between: -1. Writing an event to the event store -2. The read model/projection processing that event +In an event sourced system, there is a delay between an event being committed to the event log and a read model finishing processing it. This creates a race condition: a client performs a mutation, then immediately queries, and gets a response that does not yet reflect that mutation. -This creates a race condition where a client performs a mutation but immediately queries and gets stale data (doesn't see their own write). +Different queries tolerate this race differently. A dashboard does not care. A confirmation page shown right after a mutation does. An authorization check before a destructive action cannot tolerate it at all. The `Consistency` message lets each query state which of those situations it is in. -## Solution +## Versions in event sourcing -The `Consistency` message allows clients to specify version requirements for queries, enabling the server to retry until the projection catches up. +An event sourced system exposes distinct counters, and the `Consistency` message is only meaningful once the difference between them is clear. -## Message Definitions +The **stream version** is the position within one aggregate's event stream. It increases monotonically for that stream only. The **global position** is the position within the all-events log. It increases monotonically across every stream in the system. -### Consistency +A **projection checkpoint** is the position a read model has processed up to. A projection built from a single stream (for example, a projection that only ever reads one aggregate's events) checkpoints on stream version. A projection built from many streams (for example, a list view, a search index, or any cross-aggregate read model) checkpoints on global position. -Main message with two modes: +This gives a rule that every service using `Consistency` must follow: the `version` a client passes in `min_version` or `exact_version` must be the same counter the target projection checkpoints on. A service must document, per query, which counter it expects. The mutation response that hands the client a version must return that same counter, not the other one. Passing a stream version to a query whose projection checkpoints on global position (or the reverse) produces a request that can never be satisfied, or one that is satisfied by the wrong events. -```protobuf -message Consistency { - oneof requirement { - MinVersion min_version = 1; - ExactVersion exact_version = 2; - } - - optional google.protobuf.Duration timeout_duration = 3; - optional google.protobuf.Duration delay_duration = 4; -} -``` +The `version` field is `int64`, deliberately. It is not an opaque token; it is the actual event sourcing position, and a client is free to compare two versions or reason about their ordering. This is unlike SpiceDB's `ZedToken`, which is opaque by design and only usable as an input to a later request, never inspected or compared by the client. -### MinVersion (Recommended) +## Modes -Wait for projection to reach **at least** the specified version (`version >= X`). +### minimize_latency -**Use for**: Read-after-write consistency (most common use case) +Also the behavior when `requirement` is left unset entirely. -```protobuf -message MinVersion { - int64 version = 1; -} -``` +- **Guarantee**: the response reflects some consistent prefix of the event log. No freshness bound is promised. The prefix is always a single position; a response never mixes data from two different points in the log. +- **Input**: none. +- **Waits**: never. `timeout_duration` and `delay_duration` are ignored. +- **Cache allowed**: yes, and expected. A server may intentionally serve a slightly older position so that concurrent callers share one cached result instead of each recomputing it. +- **Output**: the result. The server should echo the position it served from, so the client can later issue a `min_version` request against that same position if it needs to confirm freshness. +- **Failure**: none specific to this mode. It cannot time out and nothing it returns can expire. +- **When to use**: browsing, listing, dashboards, and any read where staleness is acceptable. -### ExactVersion (Strict) +Setting `minimize_latency` explicitly behaves exactly like leaving `requirement` unset. The explicit form exists so the intent is visible in code review, in logs, and in traces, rather than looking like a caller who forgot to set a mode. -Wait for projection to reach **exactly** the specified version (`version == X`). +### min_version (read-your-writes) -**Use for**: Reproducible queries, audits, point-in-time reports +- **Guarantee**: the response reflects every event up to and including `version`, and possibly later events too. +- **Input**: `version`, taken from a previous mutation response. +- **Waits**: yes, until the projection checkpoint is greater than or equal to `version`, bounded by `timeout_duration` and retried every `delay_duration`. +- **Cache allowed**: only if the cached result's position is greater than or equal to `version`. +- **Output**: the result. +- **Failure**: `UNAVAILABLE` if `timeout_duration` elapses before the projection catches up. +- **When to use**: immediately after a mutation performed by the same client, to read back what was just written. This is the common mode. -```protobuf -message ExactVersion { - int64 version = 1; -} -``` +### exact_version (point-in-time snapshot) + +- **Guarantee**: the response reflects exactly the events up to `version`, and nothing after it. +- **Input**: `version`. +- **Waits**: yes, until the projection checkpoint equals `version`, bounded and retried the same way as `min_version`. +- **Cache allowed**: only if the cached result is for exactly that position, not a later one. +- **Output**: the result. +- **Failure**: `UNAVAILABLE` if `timeout_duration` elapses. `FAILED_PRECONDITION` if the projection checkpoint has already moved past `version` and the server cannot reconstruct state as of that position (the snapshot has expired). +- **When to use**: multiple reads that must agree with each other, such as paginating a report, computing a diff between two calls, or an audit. Not a tool for freshness. Passing the latest known version to `exact_version` to mean "give me the newest data" is a mistake: the request fails the moment any other write lands, because the projection has then moved past the requested version. + +### fully_consistent (head of log) + +- **Guarantee**: the response reflects every event committed to the event log before the request arrived at the server. The server, not the client, determines that head position by observing the log at the moment the request arrives. +- **Input**: none. A client that holds a version should use `min_version` instead; `fully_consistent` is for a client that has no version to offer. +- **Waits**: yes, until the projection checkpoint reaches the head position observed at request arrival, bounded by `timeout_duration` and retried every `delay_duration`. Alternatively, when a query targets a single stream, the server may answer directly from that stream's current state instead of waiting on a projection. +- **Cache allowed**: no. Any result that may predate the request's arrival at the server must not be served. +- **Output**: the result. The server should echo the head position it satisfied, so the caller can downgrade to `min_version` on a follow-up read instead of paying for `fully_consistent` again. +- **Failure**: `UNAVAILABLE` if `timeout_duration` elapses. Servers may clamp `timeout_duration` tighter for this mode, and may rate limit it, because it is the most expensive mode to satisfy. +- **When to use**: decisions where any staleness is unacceptable and the caller holds no version to pin to, such as an authorization check immediately before a destructive action, or a balance check immediately before a withdrawal. This mode should be rare. + +## Comparison + +| | minimize_latency | min_version | exact_version | fully_consistent | +|---|---|---|---|---| +| Input version | none | required | required | none | +| Waits | never | yes, bounded | yes, bounded | yes, bounded | +| Newer data than target accepted | n/a | yes | no (fails) | n/a, target is observed at request time | +| Cache allowed | yes, expected | only if position >= version | only if position == version | no | +| Failure | none | UNAVAILABLE on timeout | UNAVAILABLE on timeout, FAILED_PRECONDITION on expired snapshot | UNAVAILABLE on timeout | +| Relative cost | lowest | low | low | highest | +| Typical use | browsing, dashboards | read-your-writes after a mutation | reports, audits, repeatable reads | pre-destructive-action checks | ## Usage Pattern @@ -66,7 +88,7 @@ message ExactVersion { // Example mutation response message CreateOrderResponse { string order_id = 1; - uint64 stream_version = 2; // ← Returns current event stream version + uint64 stream_version = 2; // Returns current event stream version } ``` @@ -82,48 +104,40 @@ message GetOrderRequest { ### 3. Client Flow -1. Perform mutation and capture `stream_version` from response (e.g., version 5) -2. Immediately query with consistency requirement: - - Set `min_version.version` to the captured version - - Configure `timeout_duration` (e.g., 1s) - - Configure `delay_duration` (e.g., 100ms) -3. Server retries until projection catches up or timeout -4. Client receives result guaranteed to include their write - -## Consistency Modes Comparison - -| Feature | MinVersion | ExactVersion | -|---------|-----------|--------------| -| **Read-Your-Writes** | Yes | Yes | -| **Returns Newer Data** | Yes | Fails (FAILED_PRECONDITION) | -| **Use Case** | Normal queries | Audits, reports | -| **Flexibility** | High | Low | -| **Common** | 99% of cases | 1% of cases | +1. Perform a mutation and capture the version from the response (for example, version 5), using the same counter the target query's projection checkpoints on. +2. Immediately query with a consistency requirement: + - Set `min_version.version` to the captured version for read-your-writes. + - Set `exact_version.version` for a repeatable, point-in-time read. + - Set `fully_consistent {}` when no version is held and staleness cannot be tolerated. + - Leave `requirement` unset, or set `minimize_latency {}`, when staleness is acceptable. + - Configure `timeout_duration` (for example, 1s) and `delay_duration` (for example, 100ms) for any waiting mode. +3. Server retries until the required position is reached or the timeout elapses. +4. Client receives a result that satisfies the requested guarantee. ## Server Implementation Guidelines ### Recommended Limits -Servers should enforce reasonable limits: +Servers should enforce reasonable limits on the waiting modes (`min_version`, `exact_version`, `fully_consistent`): - **timeout_duration**: - Default: 1s (if not provided) - - Maximum: 5s (clamp values above) - + - Maximum: 5s (clamp values above, and consider a tighter maximum for `fully_consistent`) + - **delay_duration**: - Default: 100ms (if not provided) - Maximum: 500ms (clamp values above) ### Error Handling -When projection fails to meet consistency requirements, return appropriate gRPC status codes: +When a projection fails to meet a consistency requirement, return the appropriate gRPC status code: -**Projection Timeout** (timeout exceeded): +**Projection Timeout** (`min_version`, `exact_version`, or `fully_consistent`, timeout exceeded): - Status Code: `UNAVAILABLE` (503) - Use `google.rpc.ErrorInfo` for structured error details -- Suggested metadata: `minVersion`, `currentVersion`, `attempts`, `elapsedMs` +- Suggested metadata: `requiredVersion` (or `requiredHeadPosition` for `fully_consistent`), `currentVersion`, `attempts`, `elapsedMs` -**Snapshot Expired** (ExactVersion only, version moved past): +**Snapshot Expired** (`exact_version` only, projection moved past the requested version): - Status Code: `FAILED_PRECONDITION` (400) - Use `google.rpc.ErrorInfo` for structured error details - Suggested metadata: `requestedVersion`, `currentVersion` @@ -132,19 +146,28 @@ Consider using `google.rpc.Status` with `google.rpc.ErrorInfo` for rich error re ### Retry Logic +For the waiting modes: + ``` -1. Execute query -2. Check projection version -3. If version requirement met → return result -4. If ExactVersion and version > required → return FAILED_PRECONDITION -5. If timeout exceeded → return UNAVAILABLE -6. Sleep for delay_duration -7. Goto step 1 +1. Determine required position: + - min_version: required = requested version + - exact_version: required = requested version + - fully_consistent: observe log head H at request arrival; required = H +2. Execute query +3. Check projection checkpoint against required +4. If min_version or fully_consistent and checkpoint >= required -> return result +5. If exact_version and checkpoint == required -> return result +6. If exact_version and checkpoint > required -> return FAILED_PRECONDITION +7. If timeout exceeded -> return UNAVAILABLE +8. Sleep for delay_duration +9. Goto step 2 ``` +`minimize_latency` skips this loop entirely: it executes the query once against whatever the projection currently reflects and returns immediately. + ### Special Case: NOT_FOUND Errors -If query returns NOT_FOUND (entity doesn't exist in projection yet), treat it as projection lag and retry until timeout. The entity might appear once projection catches up. +If a query returns `NOT_FOUND` under `min_version`, `exact_version`, or `fully_consistent` (the entity does not yet exist in the projection), treat it as projection lag and retry until timeout rather than failing immediately. The entity may appear once the projection catches up to the required position. ## Observability @@ -153,61 +176,63 @@ If query returns NOT_FOUND (entity doesn't exist in projection yet), treat it as Track these metrics: - `consistency.attempts` - Distribution of retry attempts -- `consistency.latency` - Time to satisfy consistency requirement +- `consistency.latency` - Time to satisfy the consistency requirement - `consistency.timeouts` - Count of timeouts -- `consistency.snapshot_expired` - Count of ExactVersion failures +- `consistency.snapshot_expired` - Count of `exact_version` failures ### Logging Log when: - Clamping client-provided timeout/delay values - Timeout exceeded -- Snapshot expired (ExactVersion) +- Snapshot expired (`exact_version`) - Retry attempts (at debug level) ### Tracing Add span attributes: -- `consistency.mode` - "min_version" or "exact_version" -- `consistency.required_version` - Requested version -- `consistency.actual_version` - Projection version when satisfied -- `consistency.attempts` - Number of retries -- `consistency.elapsed_ms` - Time taken +- `consistency.mode` - one of `minimize_latency`, `min_version`, `exact_version`, `fully_consistent` +- `consistency.required_version` - requested version, for `min_version` and `exact_version` +- `consistency.observed_head` - head position observed at request arrival, for `fully_consistent` +- `consistency.actual_version` - projection version when satisfied +- `consistency.attempts` - number of retries +- `consistency.elapsed_ms` - time taken ## Trade-offs ### Pros -- Enables read-after-write consistency -- No changes to event store required -- Client controls consistency vs latency trade-off +- Lets each query state its own freshness requirement instead of a single global setting +- No changes to the event store required +- Client controls the consistency versus latency trade-off per request - Works with any projection technology ### Cons -- Increases query latency (retry loops) -- Higher server load (repeated queries) -- Client must track and pass versions -- Doesn't work with projections that don't track version +- The waiting modes increase query latency through retry loops +- The waiting modes increase server load through repeated queries +- `min_version` and `exact_version` require the client to track and pass versions +- `fully_consistent` is the most expensive mode and needs its own limits +- Does not work with projections that do not track a position ## When to Use -**Use Consistency for:** -- Mutations followed by immediate reads (e.g., "Create order → View order") -- User flows requiring strong consistency UX -- Testing/demos where eventual consistency is confusing +**minimize_latency**: browsing, listing, dashboards, anywhere staleness is acceptable. + +**min_version**: right after a mutation, to read back what was just written. + +**exact_version**: repeatable reads that must agree with each other, such as paginated reports or audits. Not for freshness. -**Don't use Consistency for:** -- Browsing/listing queries (eventual consistency is fine) -- Public-facing queries (projection lag usually acceptable) -- High-throughput read-heavy workloads (adds latency) +**fully_consistent**: decisions where no staleness is acceptable and no version is held, such as an authorization check before a destructive action. Use sparingly. + +**Avoid any waiting mode for**: high-throughput, read-heavy workloads where the added latency and server load outweigh the freshness benefit. ## Reference Inspired by: -- **SpiceDB**: `at_least_as_fresh` and `at_exact_snapshot` modes +- **SpiceDB**: `minimize_latency`, `at_least_as_fresh`, `at_exact_snapshot`, and `fully_consistent` modes - https://authzed.com/docs/spicedb/concepts/consistency - + - **DynamoDB**: Consistent reads - https://docs.aws.amazon.com/amazondynamodb/latest/developerguide/HowItWorks.ReadConsistency.html - + - **Cassandra**: QUORUM/ALL consistency levels - https://cassandra.apache.org/doc/latest/cassandra/architecture/dynamo.html#consistency-levels diff --git a/proto/trogon/consistency/v1alpha1/consistency.proto b/proto/trogon/consistency/v1alpha1/consistency.proto index 6b4d7fb..1c3d771 100644 --- a/proto/trogon/consistency/v1alpha1/consistency.proto +++ b/proto/trogon/consistency/v1alpha1/consistency.proto @@ -7,14 +7,21 @@ import "google/protobuf/duration.proto"; option (elixirpb.file).module_prefix = "TrogonProto.Consistency.V1Alpha1"; -// Configuration for read-your-writes consistency guarantees in eventual consistency systems. +// Configuration for consistency guarantees in eventual consistency systems. // -// Used in query operations to wait for projections/read models to catch up to a specific version -// after a write operation. This enables clients to immediately read their own writes despite -// projection lag in CQRS/Event Sourcing architectures. +// Used in query operations to control how a query relates to the position of +// the event log, from accepting whatever a projection currently reflects, to +// waiting for a specific position, to waiting for the head of the log as +// observed at request time. // // ## Consistency Modes // +// ### MinimizeLatency (Default - No Freshness Guarantee) +// Returns whatever a projection currently reflects, with no wait and no freshness +// bound. Serving a cached or slightly stale result is expected. This is also what +// happens when `requirement` is left unset; the explicit form exists so intent is +// visible in code, logs, and tracing. +// // ### MinVersion (Recommended - Read-Your-Writes) // Waits for projection to reach AT LEAST the specified version. Will use newer data if available. // Use this after mutations to ensure you can immediately read what you just wrote. @@ -23,6 +30,11 @@ option (elixirpb.file).module_prefix = "TrogonProto.Consistency.V1Alpha1"; // Waits for projection to reach EXACTLY the specified version. Rejects if version is newer. // Use this only when you need reproducible queries at a specific point-in-time (audits, reports). // +// ### FullyConsistent (Strongest - Head of Log) +// Waits for projection to reach the position of the event log observed by the server +// at the moment the request arrived. Use this when a caller holds no version but any +// staleness is unacceptable. +// // ## Usage Pattern // // 1. Client performs mutation (e.g., CreateOrder, UpdateInventory) @@ -30,7 +42,8 @@ option (elixirpb.file).module_prefix = "TrogonProto.Consistency.V1Alpha1"; // 3. Client immediately queries with Consistency: // - min_version { version: 5 } (recommended - read-your-writes) // - exact_version { version: 5 } (strict - reproducible snapshot) -// 4. Server retries query until projection reaches version or timeout +// - fully_consistent {} (strongest - no version held by the client) +// 4. Server retries query until projection reaches the required position or timeout // // ## Example // @@ -68,11 +81,18 @@ option (elixirpb.file).module_prefix = "TrogonProto.Consistency.V1Alpha1"; // // ## Reference // -// Inspired by SpiceDB's Consistency patterns (at_least_as_fresh and at_exact_snapshot modes). +// Inspired by SpiceDB's Consistency patterns. SpiceDB has four comparable modes: +// minimize_latency, at_least_as_fresh, at_exact_snapshot, and fully_consistent. // See: https://authzed.com/docs/spicedb/concepts/consistency message Consistency { - // Which consistency mode to use + // Which consistency mode to use. Leaving this unset is equivalent to minimize_latency. oneof requirement { + // Accept whatever position a projection currently reflects. No wait, no freshness bound. + // + // This is also the behavior when requirement is left unset entirely. Set it explicitly + // when the intent should be visible in code, logs, and tracing. + MinimizeLatency minimize_latency = 5; + // Wait for projection to reach AT LEAST this version (version >= X). // // Recommended for read-your-writes consistency. If projection has already @@ -90,12 +110,19 @@ message Consistency { // // Use only for: audit queries, reports requiring exact historical state. ExactVersion exact_version = 2; + + // Wait for projection to reach the position of the event log observed by the + // server at the moment the request arrived. + // + // Use when the caller holds no version and any staleness is unacceptable. + // If the caller does hold a version, prefer min_version instead. + FullyConsistent fully_consistent = 6; } // Maximum time to wait for projection to catch up. // - // Applies to any requirement mode above. If requirement is not set (eventual consistency), - // this field is ignored. + // Applies to any requirement mode above. If requirement is not set, or is + // minimize_latency, this field is ignored. // // Recommended server limits: // - Default: 1s (if not provided) @@ -107,8 +134,8 @@ message Consistency { // Delay between retry attempts. // - // Applies to any requirement mode above. If requirement is not set (eventual consistency), - // this field is ignored. + // Applies to any requirement mode above. If requirement is not set, or is + // minimize_latency, this field is ignored. // // Recommended server limits: // - Default: 100ms (if not provided) @@ -119,6 +146,9 @@ message Consistency { optional google.protobuf.Duration delay_duration = 4; } +// Accept whatever position a projection currently reflects, with no freshness bound. +message MinimizeLatency {} + // Wait for projection to be at least as fresh as the specified version. message MinVersion { // Minimum stream version required (projection.version >= this). @@ -136,3 +166,7 @@ message ExactVersion { // failed precondition error if projection has moved past this version. int64 version = 1; } + +// Wait for projection to reach the position of the event log observed by the +// server at the moment the request arrived. +message FullyConsistent {} From 23ad41d07355b3b5913259fbfb29df5cc7388d6c Mon Sep 17 00:00:00 2001 From: Yordis Prieto Date: Wed, 7 Oct 2026 03:03:23 -0400 Subject: [PATCH 2/3] docs(consistency): validate the result position instead of a later checkpoint Signed-off-by: Yordis Prieto --- docs/explanation/consistency-pattern.md | 19 +++++++++++++------ 1 file changed, 13 insertions(+), 6 deletions(-) diff --git a/docs/explanation/consistency-pattern.md b/docs/explanation/consistency-pattern.md index 9fd4219..647f85c 100644 --- a/docs/explanation/consistency-pattern.md +++ b/docs/explanation/consistency-pattern.md @@ -153,21 +153,28 @@ For the waiting modes: - min_version: required = requested version - exact_version: required = requested version - fully_consistent: observe log head H at request arrival; required = H -2. Execute query -3. Check projection checkpoint against required -4. If min_version or fully_consistent and checkpoint >= required -> return result -5. If exact_version and checkpoint == required -> return result -6. If exact_version and checkpoint > required -> return FAILED_PRECONDITION +2. Execute the query and capture the position the result reflects. + The result and its position must come from the same projection snapshot + (for example, the same read transaction, or the same cached entry). +3. Compare that result position, not a later checkpoint, with required. +4. If min_version or fully_consistent and result position >= required -> return result +5. If exact_version and result position == required -> return result +6. If exact_version and result position > required -> return FAILED_PRECONDITION 7. If timeout exceeded -> return UNAVAILABLE 8. Sleep for delay_duration 9. Goto step 2 ``` +The position must be read together with the result. Reading the projection checkpoint after the query is a race: the projection can advance between the query and the checkpoint read, so a checkpoint that satisfies `required` says nothing about the rows that were returned. Under `exact_version` the same race can return rows from before the requested position while the checkpoint reads as equal to it. Every projection that supports a waiting mode therefore needs a way to return its position atomically with the query result. + `minimize_latency` skips this loop entirely: it executes the query once against whatever the projection currently reflects and returns immediately. ### Special Case: NOT_FOUND Errors -If a query returns `NOT_FOUND` under `min_version`, `exact_version`, or `fully_consistent` (the entity does not yet exist in the projection), treat it as projection lag and retry until timeout rather than failing immediately. The entity may appear once the projection catches up to the required position. +Under `min_version`, `exact_version`, or `fully_consistent`, a `NOT_FOUND` result carries a position like any other result, and the same comparison applies to it: + +- If the result position is below `required`, the entity may not exist yet because of projection lag. Treat it as not satisfied and retry until timeout. +- If the result position satisfies `required`, the entity genuinely does not exist at that position. Return `NOT_FOUND`. Retrying would only convert a correct answer into `UNAVAILABLE` after the timeout elapses. ## Observability From 99e641aef91ff4e8598f1eaaefed5958fc92ba8c Mon Sep 17 00:00:00 2001 From: Yordis Prieto Date: Wed, 7 Oct 2026 03:13:14 -0400 Subject: [PATCH 3/3] docs(consistency): explain where the version comes from on the write side Signed-off-by: Yordis Prieto --- docs/explanation/consistency-pattern.md | 29 +++++++++++++++---- .../consistency/v1alpha1/consistency.proto | 13 +++++---- 2 files changed, 32 insertions(+), 10 deletions(-) diff --git a/docs/explanation/consistency-pattern.md b/docs/explanation/consistency-pattern.md index 647f85c..564f384 100644 --- a/docs/explanation/consistency-pattern.md +++ b/docs/explanation/consistency-pattern.md @@ -20,6 +20,25 @@ A **projection checkpoint** is the position a read model has processed up to. A This gives a rule that every service using `Consistency` must follow: the `version` a client passes in `min_version` or `exact_version` must be the same counter the target projection checkpoints on. A service must document, per query, which counter it expects. The mutation response that hands the client a version must return that same counter, not the other one. Passing a stream version to a query whose projection checkpoints on global position (or the reverse) produces a request that can never be satisfied, or one that is satisfied by the wrong events. +### Where the version comes from + +The write side has to hand the client the counter the target projection checkpoints on, and how that counter becomes available depends on what the event store returns when a stream is appended to. + +Some stores return every counter on append. EventStoreDB's append result carries the stream revision of the last appended event and its position in the `$all` log, so a mutation handler can return either without a second read. Each event delivered by a subscription carries the same pair, and a projection records whichever one it consumes: a catch-up subscription to `$all` checkpoints on the log position, a subscription to a single stream checkpoints on the stream revision. When the store exposes the log position as a composite (EventStoreDB's commit and prepare positions), the commit position is the usual choice for the `int64` value, with the checkpoint recorded once every event sharing that commit position has been processed. + +Some stores return only the stream version on append. Commanded's dispatch result exposes the aggregate version, while the global event number appears only on the recorded events later delivered to event handlers and projectors. Commanded's Ecto projections checkpoint on that global event number. A mutation handler that needs it reads the appended events back from the aggregate's stream and takes the global number of the last one, which is one extra read per mutation taken on the write path. + +The translation can instead happen at query time. The mutation returns the stream version, and the query handler locates that event in its stream to learn the global position before comparing with the checkpoint. This keeps the write path free of extra reads at the cost of one read per consistent query, and it lets the API contract speak only in stream versions, which is simpler for clients. + +Principles that hold regardless of store: + +- Whatever counter crosses the API boundary, the server must be able to compare it with the projection checkpoint without guessing. Document which counter each mutation returns and which counter each query expects. +- Prefer returning the counter the projection checkpoints on directly when the store provides it on append. +- When the store provides only the stream version, choose one translation point per service, write path or query path, and keep it consistent. +- The counter is a log position, not a clock. Never derive it from timestamps, and never compare counters from different stores or different event logs. + +Some frameworks also offer a write-side alternative. Commanded can block a dispatch until strongly consistent handlers have processed the new events, which removes projection lag for the caller at the cost of slower writes. This package expresses the same need on the read side instead, so that each query pays only for the consistency it asks for. A service may use either, but should not wait on both sides for the same projection. + The `version` field is `int64`, deliberately. It is not an opaque token; it is the actual event sourcing position, and a client is free to compare two versions or reason about their ordering. This is unlike SpiceDB's `ZedToken`, which is opaque by design and only usable as an input to a later request, never inspected or compared by the client. ## Modes @@ -42,7 +61,7 @@ Setting `minimize_latency` explicitly behaves exactly like leaving `requirement` - **Guarantee**: the response reflects every event up to and including `version`, and possibly later events too. - **Input**: `version`, taken from a previous mutation response. -- **Waits**: yes, until the projection checkpoint is greater than or equal to `version`, bounded by `timeout_duration` and retried every `delay_duration`. +- **Waits**: yes, until a result reflecting a position greater than or equal to `version` is available, bounded by `timeout_duration` and retried every `delay_duration`. - **Cache allowed**: only if the cached result's position is greater than or equal to `version`. - **Output**: the result. - **Failure**: `UNAVAILABLE` if `timeout_duration` elapses before the projection catches up. @@ -52,17 +71,17 @@ Setting `minimize_latency` explicitly behaves exactly like leaving `requirement` - **Guarantee**: the response reflects exactly the events up to `version`, and nothing after it. - **Input**: `version`. -- **Waits**: yes, until the projection checkpoint equals `version`, bounded and retried the same way as `min_version`. +- **Waits**: yes, until a result reflecting exactly `version` is available, bounded and retried the same way as `min_version`. - **Cache allowed**: only if the cached result is for exactly that position, not a later one. - **Output**: the result. -- **Failure**: `UNAVAILABLE` if `timeout_duration` elapses. `FAILED_PRECONDITION` if the projection checkpoint has already moved past `version` and the server cannot reconstruct state as of that position (the snapshot has expired). +- **Failure**: `UNAVAILABLE` if `timeout_duration` elapses. `FAILED_PRECONDITION` if the projection has already moved past `version` and the server cannot reconstruct state as of that position (the snapshot has expired). - **When to use**: multiple reads that must agree with each other, such as paginating a report, computing a diff between two calls, or an audit. Not a tool for freshness. Passing the latest known version to `exact_version` to mean "give me the newest data" is a mistake: the request fails the moment any other write lands, because the projection has then moved past the requested version. ### fully_consistent (head of log) - **Guarantee**: the response reflects every event committed to the event log before the request arrived at the server. The server, not the client, determines that head position by observing the log at the moment the request arrives. - **Input**: none. A client that holds a version should use `min_version` instead; `fully_consistent` is for a client that has no version to offer. -- **Waits**: yes, until the projection checkpoint reaches the head position observed at request arrival, bounded by `timeout_duration` and retried every `delay_duration`. Alternatively, when a query targets a single stream, the server may answer directly from that stream's current state instead of waiting on a projection. +- **Waits**: yes, until a result reflecting a position at or past the head observed at request arrival is available, bounded by `timeout_duration` and retried every `delay_duration`. Alternatively, when a query targets a single stream, the server may answer directly from that stream's current state instead of waiting on a projection. - **Cache allowed**: no. Any result that may predate the request's arrival at the server must not be served. - **Output**: the result. The server should echo the head position it satisfied, so the caller can downgrade to `min_version` on a follow-up read instead of paying for `fully_consistent` again. - **Failure**: `UNAVAILABLE` if `timeout_duration` elapses. Servers may clamp `timeout_duration` tighter for this mode, and may rate limit it, because it is the most expensive mode to satisfy. @@ -88,7 +107,7 @@ Setting `minimize_latency` explicitly behaves exactly like leaving `requirement` // Example mutation response message CreateOrderResponse { string order_id = 1; - uint64 stream_version = 2; // Returns current event stream version + int64 version = 2; // The counter the target projection checkpoints on } ``` diff --git a/proto/trogon/consistency/v1alpha1/consistency.proto b/proto/trogon/consistency/v1alpha1/consistency.proto index 1c3d771..70b042b 100644 --- a/proto/trogon/consistency/v1alpha1/consistency.proto +++ b/proto/trogon/consistency/v1alpha1/consistency.proto @@ -38,7 +38,7 @@ option (elixirpb.file).module_prefix = "TrogonProto.Consistency.V1Alpha1"; // ## Usage Pattern // // 1. Client performs mutation (e.g., CreateOrder, UpdateInventory) -// 2. Server returns stream_version (e.g., version 5) +// 2. Server returns the version (e.g., version 5), using the counter the target projection checkpoints on // 3. Client immediately queries with Consistency: // - min_version { version: 5 } (recommended - read-your-writes) // - exact_version { version: 5 } (strict - reproducible snapshot) @@ -51,7 +51,7 @@ option (elixirpb.file).module_prefix = "TrogonProto.Consistency.V1Alpha1"; // // Mutation response // message CreateOrderResponse { // string order_id = 1; -// uint64 stream_version = 2; // Returns: 5 +// int64 version = 2; // Returns: 5, the counter the target projection checkpoints on // } // // // Read-your-writes (recommended) @@ -151,17 +151,20 @@ message MinimizeLatency {} // Wait for projection to be at least as fresh as the specified version. message MinVersion { - // Minimum stream version required (projection.version >= this). + // Minimum position required (projection.position >= this). // - // Client should use the stream_version returned from their previous mutation. + // Either a stream version or a global log position, whichever counter the + // target projection checkpoints on. Client should use the version returned + // from their previous mutation. // Server should retry query until projection reaches or exceeds this version. int64 version = 1; } // Wait for projection to reach exactly the specified version (strict snapshot). message ExactVersion { - // Exact stream version required (projection.version == this). + // Exact position required (projection.position == this). // + // Same counter rule as MinVersion. // Server should retry until projection reaches this version, or fail with // failed precondition error if projection has moved past this version. int64 version = 1;