diff --git a/Cargo.lock b/Cargo.lock index b695c61..e10c7f3 100644 --- a/Cargo.lock +++ b/Cargo.lock @@ -2442,6 +2442,7 @@ version = "0.209.0" dependencies = [ "aws-sdk-dynamodb", "futures-util", + "inventory", "nexus-chat", "nexus-flow-core", "nexus-flow-facade", diff --git a/changes/1c5s-server-derived-keys.md b/changes/1c5s-server-derived-keys.md new file mode 100644 index 0000000..cd0fa3a --- /dev/null +++ b/changes/1c5s-server-derived-keys.md @@ -0,0 +1,8 @@ +--- +type: added +facade: changed +--- +[en] +A hosted board now ranks `next` by derived ranking keys (`related`, `count`, `sum`) exactly as a local replica does. The store-free `next` (`read::next_active`, `next_active_value`, `next_active_filtered`) ranks a plugin with such keys from a new `ActiveBoard::ranking` graph: the edges around the active tickets and the live neighbours they reach, closed and archived ones included, deleted ones not. A server fills it with `nxs_fold_ddb::records::with_ranking` for what `read::next_ranking_reads(cfg)` names; `read::with_ranking` reads the same graph from a store. Without the graph, `next_active` still refuses such a plugin. Everything is additive: `ActiveBoard` gains a field (it is `#[non_exhaustive]`), and `RankingReads`, `RankingGraph`, `Neighbour`, `next_ranking_reads` and `with_ranking` are new. For a server's operators: the fold's index revision goes from 2 to 3, so every hosted stream is not current and is folded again on its next run. Its records reads (`active_board`, `with_ranking`, `list_board`) answer `NotCurrent` from the moment the run clears it and `Refolding` until the refold has reached the relay position the clear named — never a half-folded board. `Folder::clear` now takes that position (`clear(table, until)`) and leaves a sentinel watermark instead of deleting it. Roll the new build out to every fold run at once: a build with older rules refolds a newer stream back. The `.adj#` adjacency entries gain attributes; the one table-contract change of this release, the `typed` index, is described with the hosted `list`. +[de] +Ein gehostetes Brett rankt `next` jetzt nach abgeleiteten Ranking-Schlüsseln (`related`, `count`, `sum`) genau so wie ein lokales Replikat. Das speicherfreie `next` (`read::next_active`, `next_active_value`, `next_active_filtered`) rankt ein Plugin mit solchen Schlüsseln aus einem neuen Graphen `ActiveBoard::ranking`: den Kanten um die aktiven Tickets und den lebenden Nachbarn, die sie erreichen, geschlossene und archivierte eingeschlossen, gelöschte nicht. Ein Server füllt ihn mit `nxs_fold_ddb::records::with_ranking` für das, was `read::next_ranking_reads(cfg)` nennt; `read::with_ranking` liest denselben Graphen aus einem Speicher. Ohne den Graphen lehnt `next_active` ein solches Plugin weiterhin ab. Alles ist additiv: `ActiveBoard` bekommt ein Feld (es ist `#[non_exhaustive]`), und `RankingReads`, `RankingGraph`, `Neighbour`, `next_ranking_reads` und `with_ranking` sind neu. Für den Betrieb eines Servers: Die Index-Revision des Folds steigt von 2 auf 3, daher ist jeder gehostete Stream nicht aktuell und wird bei seinem nächsten Lauf neu gefaltet. Seine Records-Lesezugriffe (`active_board`, `with_ranking`, `list_board`) antworten ab dem Moment, in dem der Lauf ihn leert, mit `NotCurrent` und mit `Refolding`, bis das Neufalten die Relay-Position erreicht hat, die das Leeren genannt hat — nie mit einem halb gefalteten Brett. `Folder::clear` nimmt diese Position jetzt entgegen (`clear(table, until)`) und hinterlässt ein Sentinel-Watermark, statt es zu löschen. Den neuen Build in alle Fold-Läufe zugleich ausrollen: Ein Build mit älteren Regeln faltet einen neueren Stream zurück. Die `.adj#`-Adjazenzeinträge bekommen Attribute; die eine Änderung des Tabellenvertrags in diesem Release, der Index `typed`, ist bei der gehosteten `list` beschrieben. diff --git a/changes/xkr7-typed-index.md b/changes/xkr7-typed-index.md new file mode 100644 index 0000000..281a6b5 --- /dev/null +++ b/changes/xkr7-typed-index.md @@ -0,0 +1,8 @@ +--- +type: added +facade: changed +--- +[en] +A hosted board now answers `list` ranked by the plugin's `[ranking.list]`, derived ranking keys included, exactly as a local replica does. The facade names what to read for a query (`read::list_reads`, a new `ListReads`), a server reads it into a new `ListBoard` with `nxs_fold_ddb::records::list_board`, and `read::list_board_page` / `read::list_board_value` filter (type, container, label, status), rank and project it with the code `list` runs locally; `read::list_board` reads the same board from a store. The hosted list is not paged: `limit` cuts it with the total disclosed, and `paginate` or a token is refused, as for the hosted `next`. A list that is not ranked costs its page, not the stream: `ListReads::select` (a new `ListSelect`) lets the server keep and cut on the item rows before it reads any join (`read::list_select`, and `ListBoard::cut` with a new `ListCut` carries the total counted before the cut), and the server reads its point reads by `BatchGetItem`; a ranked list reads its types' whole `list` scope, which its rank is computed over. A hosted `next` whose keys read the rank `[ranking.list]` gives a type now ranks too instead of being refused: `RankingReads` gains `list_types` and `RankingGraph` gains `listed` (both `#[non_exhaustive]`, so additive). For a server's operators, the table contract changes: a third sparse index, GSI `typed` — partition key `nxf_typed` (string, `#`), sort key `sk` (string), projection ALL — set by the fold on every live item row (closed and archived included, removed on delete). Add the index to the stack first, then deploy the read; a fold against a table without the index writes the attribute harmlessly. It ships in fold index revision 3 together with the self-describing adjacency entries, so one refold covers both. For an embedder of `nxs-fold-ddb`: the `Table` trait gains `get_many` (one `BatchGetItem` per 100 keys, unprocessed keys sent again), `RecordsError` gains `Refolding`, `Watermark` gains `refold_until`, `board::reindex` returns the items whose index attributes did not fit, and an item op is refused (`Folded::Refused`) when its row would pass 400 KB together with its index attributes, at one `GetItem` more per op, instead of silently dropping off an index. +[de] +Ein gehostetes Brett beantwortet `list` jetzt nach dem `[ranking.list]` des Plugins gerankt, abgeleitete Ranking-Schlüssel eingeschlossen, genau so wie ein lokales Replikat. Die Fassade nennt, was für eine Anfrage zu lesen ist (`read::list_reads`, ein neues `ListReads`), ein Server liest es mit `nxs_fold_ddb::records::list_board` in ein neues `ListBoard`, und `read::list_board_page` / `read::list_board_value` filtern (Typ, Container, Label, Status), ranken und projizieren es mit dem Code, den `list` lokal ausführt; `read::list_board` liest dasselbe Brett aus einem Speicher. Die gehostete Liste wird nicht geblättert: `limit` schneidet sie mit offengelegter Gesamtzahl, `paginate` oder ein Token wird abgelehnt, wie beim gehosteten `next`. Eine nicht gerankte Liste kostet ihre Seite, nicht den Stream: `ListReads::select` (ein neues `ListSelect`) lässt den Server anhand der Item-Zeilen behalten und schneiden, bevor er einen Join liest (`read::list_select`; `ListBoard::cut` mit einem neuen `ListCut` trägt die vor dem Schnitt gezählte Gesamtzahl), und der Server liest seine Punktzugriffe per `BatchGetItem`; eine gerankte Liste liest den ganzen `list`-Bereich ihrer Typen, über den ihr Rang berechnet wird. Ein gehostetes `next`, dessen Schlüssel den Rang lesen, den `[ranking.list]` einem Typ gibt, rankt jetzt ebenfalls, statt abgelehnt zu werden: `RankingReads` bekommt `list_types` und `RankingGraph` bekommt `listed` (beide `#[non_exhaustive]`, also additiv). Für den Betrieb eines Servers ändert sich der Tabellenvertrag: ein dritter dünn besetzter Index, GSI `typed` — Partitionsschlüssel `nxf_typed` (String, `#`), Sortierschlüssel `sk` (String), Projektion ALL —, den der Fold an jeder lebenden Item-Zeile setzt (geschlossene und archivierte eingeschlossen, beim Löschen entfernt). Zuerst den Index in den Stack aufnehmen, dann das Lesen ausrollen; ein Fold gegen eine Tabelle ohne den Index schreibt das Attribut folgenlos. Er kommt mit der Fold-Index-Revision 3 zusammen mit den selbstbeschreibenden Adjazenzeinträgen, ein Neufalten deckt also beides ab. Für Einbettende von `nxs-fold-ddb`: Das Trait `Table` bekommt `get_many` (ein `BatchGetItem` je 100 Schlüssel, unverarbeitete Schlüssel werden erneut gesendet), `RecordsError` bekommt `Refolding`, `Watermark` bekommt `refold_until`, `board::reindex` gibt die Items zurück, deren Index-Attribute nicht passten, und eine Item-Op wird abgelehnt (`Folded::Refused`), wenn ihre Zeile zusammen mit ihren Index-Attributen 400 KB überschritte — zum Preis eines `GetItem` mehr je Op —, statt still aus einem Index zu fallen. diff --git a/crates/core/src/graph.rs b/crates/core/src/graph.rs index dca4ce6..34b8f84 100644 --- a/crates/core/src/graph.rs +++ b/crates/core/src/graph.rs @@ -30,7 +30,8 @@ pub struct Ticket { } /// A present edge, `from` → `to` (a `dep` points at its blocker, a `parent` edge at the parent). -#[derive(Debug, Clone, PartialEq, Eq)] +/// Ordered by `from`, `to`, `kind`, so a set of edges has one order whatever order they came in. +#[derive(Debug, Clone, PartialEq, Eq, PartialOrd, Ord)] pub struct Edge { pub from: String, pub to: String, diff --git a/crates/core/src/model.rs b/crates/core/src/model.rs index e00fb57..5beca2d 100644 --- a/crates/core/src/model.rs +++ b/crates/core/src/model.rs @@ -48,7 +48,7 @@ impl Status { /// the OR-set replacement for the former single-parent `belongs_to` LWW register. The substrate is /// n:m-capable like every other edge; single-parent is enforced at the write seam during ②a and /// relaxed in ②c. The single current parent is projected by the `present_parent` view. -#[derive(Debug, Clone, Copy, PartialEq, Eq, Hash)] +#[derive(Debug, Clone, Copy, PartialEq, Eq, PartialOrd, Ord, Hash)] pub enum EdgeKind { Dep, ContributesTo, diff --git a/crates/facade/src/read.rs b/crates/facade/src/read.rs index 47d775b..0c5da3b 100644 --- a/crates/facade/src/read.rs +++ b/crates/facade/src/read.rs @@ -24,7 +24,7 @@ use crate::plugin::{Dir, Nulls, PluginConfig, RankKey, RankSpec, Verb}; use crate::record::{item_value, items_value, raw_field}; use nexus_flow_core::derive; use nexus_flow_core::graph::Board; -use nexus_flow_core::model::{ItemRow, LinkWeight, ThreadLink}; +use nexus_flow_core::model::{EdgeKind, ItemRow, LinkWeight, ThreadLink}; use nexus_flow_core::store::Store; use serde_json::{json, Value}; use std::collections::{BTreeMap, BTreeSet, HashMap, HashSet}; @@ -2102,6 +2102,20 @@ fn blocked_record(cfg: Option<&PluginConfig>, row: &BlockedItem, j: &RecordJoins rec } +/// One `list` record without a store (6j6v.xkr7): what [`list_to_value_with_custom`] gives — the +/// canonical item, the sparse `labels` and `parent_closed_reason` joins, and the lane joins. +fn list_record(cfg: Option<&PluginConfig>, item: &ItemRow, j: &RecordJoins) -> Value { + let mut rec = item_value(item); + if !j.labels.is_empty() { + rec["labels"] = json!(j.labels); + } + if let Some(reasons) = parent_closed_reason_value(j.parent_closed) { + rec["parent_closed_reason"] = reasons; + } + put_lane_joins(cfg, &mut rec, j); + rec +} + /// One record of a bare lane (`deferred`/`closed`/`archived`/`search`): the canonical item and the /// lane joins. fn lane_record(cfg: Option<&PluginConfig>, item: &ItemRow, j: &RecordJoins) -> Value { @@ -2578,7 +2592,15 @@ fn date_cmp(a: &ItemRow, b: &ItemRow, field: &str, dir: Dir) -> std::cmp::Orderi fn sort_cmp(cfg: &PluginConfig, a: &ItemRow, b: &ItemRow, key: SortKey) -> std::cmp::Ordering { match key { SortKey::Rank => rank_cmp(cfg, a, b), - SortKey::Id => a.id.cmp(&b.id), + _ => row_cmp(a, b, key), + } +} + +/// [`sort_cmp`] for every key but `rank`: the orders that read the row alone, and no plugin. +fn row_cmp(a: &ItemRow, b: &ItemRow, key: SortKey) -> std::cmp::Ordering { + match key { + // Never asked: `sort_cmp` ranks, and a `ListSelect` never orders by rank. + SortKey::Rank | SortKey::Id => a.id.cmp(&b.id), SortKey::Defer => date_cmp(a, b, "defer_until", Dir::Asc), SortKey::Closed => date_cmp(a, b, "closed_at", Dir::Desc), SortKey::Archived => date_cmp(a, b, "archived", Dir::Desc), @@ -2734,6 +2756,11 @@ pub struct ActiveBoard { pub board: Board, /// By id. pub tickets: BTreeMap, + /// The graph the derived ranking keys of `[ranking.next]` read (6j6v.1c5s): `None` until a + /// reader fills it — [`with_ranking`] from a store, `nxs_fold_ddb::records::with_ranking` from + /// the folded tables — for what [`next_ranking_reads`] names. A plugin without derived `next` + /// keys never needs it; [`next_active`] refuses one with such keys while it is missing. + pub ranking: Option, } impl ActiveBoard { @@ -2745,6 +2772,7 @@ impl ActiveBoard { .into_iter() .map(|t| (t.item.id.clone(), t)) .collect(), + ranking: None, } } @@ -2780,6 +2808,209 @@ impl ActiveBoard { } } +/// What the derived ranking keys of a `[ranking.next]` read beyond the active tickets' records +/// (6j6v.1c5s) — what a reader of an [`ActiveBoard`] puts in its [`RankingGraph`]. +/// [`next_ranking_reads`] gives it for a plugin. +/// +/// `#[non_exhaustive]`: start from `RankingReads::default()` and set the fields. +#[derive(Debug, Clone, Default, PartialEq, Eq)] +#[non_exhaustive] +pub struct RankingReads { + /// The edge kinds the keys follow or count; `mentions` too when a key counts `mentioned_by`. + pub kinds: BTreeSet, + /// A count filters on a label: the neighbours' labels. + pub labels: bool, + /// An `of` reads a custom field: the neighbours' custom values. + pub custom: bool, + /// The types whose `[ranking.list]` rank a key reads (6j6v.xkr7): the graph also holds every + /// live, unarchived item of each — the type's `list` scope ([`RankingGraph::listed`]) — and the + /// edges around them. A server reads them from its `typed` index. + pub list_types: BTreeSet, +} + +impl RankingReads { + /// Whether a graph read for `self` holds everything a graph read for `needed` holds. + pub fn covers(&self, needed: &RankingReads) -> bool { + needed.kinds.is_subset(&self.kinds) + && (self.labels || !needed.labels) + && (self.custom || !needed.custom) + && needed.list_types.is_subset(&self.list_types) + } +} + +/// The graph around a board's rows that the derived ranking keys read (6j6v.1c5s, +/// docs/specs/derived-ranking-keys.md §6): the input [`next_active`] ranks a `[ranking.next]` with +/// `related` / `count` / `sum` keys from, and [`list_board_page`] a `[ranking.list]` (6j6v.xkr7), +/// so a server ranks exactly as [`next`] and [`list_page`] do over a store. +/// +/// The SUBJECTS are the rows a key is evaluated on: the board's own rows — an [`ActiveBoard`]'s +/// active tickets; a [`ListBoard`]'s listed items and, when it carries one, its active board's +/// tickets — and the `listed` items below. +/// +/// - `edges`: every present edge of a kind in `reads.kinds` with at least one end a subject and +/// both ends LIVE (not deleted, not missing). A `parent` edge is the `present_parent` projection — +/// a child's one current parent, the one `belongs_to` reads — so a child re-parented concurrently +/// is under one parent only. Distinct, ordered. +/// - `listed`: the ids of every live, unarchived item of a type in `reads.list_types` — the `list` +/// scope a `[ranking.list]` rank is computed over (6j6v.xkr7). Empty without list types. +/// - `neighbours`: every live row the board does not hold itself — the other ends of `edges` and +/// the `listed` items, closed and archived items included, since a key reads every LIVE neighbour +/// (owner, 2026-10-11) — each with its row (`belongs_to` filled as the store fills it), its +/// labels when `reads.labels` and its non-empty custom values when `reads.custom`, else empty. +/// +/// `#[non_exhaustive]`: build it with [`RankingGraph::new`] and set the fields. +#[derive(Debug, Clone, Default, PartialEq, Eq)] +#[non_exhaustive] +pub struct RankingGraph { + /// What the graph was read for. + pub reads: RankingReads, + pub edges: BTreeSet, + /// By id. + pub neighbours: BTreeMap, + /// The live, unarchived items of `reads.list_types` (6j6v.xkr7), by id. + pub listed: BTreeSet, +} + +impl RankingGraph { + /// An empty graph read for `reads`. + pub fn new(reads: RankingReads) -> RankingGraph { + RankingGraph { + reads, + ..RankingGraph::default() + } + } +} + +/// A live item a derived ranking key reads that the board does not hold itself — next to one of +/// its rows, or in the `list` scope of a type ([`RankingGraph::neighbours`]). +/// +/// `#[non_exhaustive]`: build it with [`Neighbour::new`] and set the fields. +#[derive(Debug, Clone, PartialEq, Eq)] +#[non_exhaustive] +pub struct Neighbour { + pub item: ItemRow, + /// Present labels, sorted and distinct, as `present_labels`. + pub labels: Vec, + /// Every non-empty custom value, as the store reads them in bulk. + pub custom: BTreeMap, +} + +impl Neighbour { + /// A neighbour with no labels or custom values read. + pub fn new(item: ItemRow) -> Neighbour { + Neighbour { + item, + labels: Vec::new(), + custom: BTreeMap::new(), + } + } +} + +/// What the store-free `next` of `cfg` reads beyond the active tickets' records — the +/// [`RankingReads`] a reader fills [`ActiveBoard::ranking`] for — or `None` when `[ranking.next]` +/// declares no derived ranking key (6j6v.1c5s). It does not depend on the board. +/// +/// A `[ranking.next]` that reads, directly or down its type chain, the rank of a type only +/// `[ranking.list]` orders names that type in [`RankingReads::list_types`] (6j6v.xkr7): that rank +/// is computed over the type's live items, which the reader puts in the graph. Until 6j6v.xkr7 +/// such a plugin was refused here; it is no longer, and the `Result` stays for a later refusal. +pub fn next_ranking_reads(cfg: &PluginConfig) -> Result> { + derived::reads_for_next(cfg) +} + +/// `board` with its [`RankingGraph`] for `reads` read from `store` (6j6v.1c5s) — the parity oracle +/// for `nxs_fold_ddb::records::with_ranking`, as [`active_board`] is for the records. One edge pass +/// ([`Store::present_edges_of_kinds`]), one bulk read of the neighbours' rows, and one of their +/// labels and of their custom values when `reads` asks for them; with list types, one read of the +/// live items (6j6v.xkr7). +pub fn with_ranking( + store: &Store, + mut board: ActiveBoard, + reads: &RankingReads, +) -> Result { + let held: BTreeSet<&str> = board.tickets.keys().map(String::as_str).collect(); + board.ranking = Some(ranking_graph(store, reads, &held)?); + Ok(board) +} + +/// The [`RankingGraph`] for `reads` around the rows a board holds itself (`held`, every one a +/// subject), read from `store` — what [`with_ranking`] and [`list_board`] put on their boards. +fn ranking_graph( + store: &Store, + reads: &RankingReads, + held: &BTreeSet<&str>, +) -> Result { + let listed: Vec = if reads.list_types.is_empty() { + Vec::new() + } else { + derived::list_scope(store)? + .into_iter() + .filter(|i| { + i.item_type + .as_deref() + .is_some_and(|t| reads.list_types.contains(t)) + }) + .collect() + }; + let listed_ids: BTreeSet<&str> = listed.iter().map(|i| i.id.as_str()).collect(); + let subject = |id: &str| held.contains(id) || listed_ids.contains(id); + let kinds: Vec = reads.kinds.iter().copied().collect(); + let touching: Vec<(String, String, EdgeKind)> = store + .present_edges_of_kinds(&kinds)? + .into_iter() + .filter(|(from, to, _)| subject(from) || subject(to)) + .collect(); + let others: BTreeSet<&str> = touching + .iter() + .flat_map(|(from, to, _)| [from.as_str(), to.as_str()]) + .filter(|id| !subject(id)) + .collect(); + let others: Vec<&str> = others.into_iter().collect(); + let live: BTreeMap = store + .get_items(&others)? + .into_iter() + .filter(|i| i.deleted.as_deref() != Some("1")) + .map(|i| (i.id.clone(), i)) + .collect(); + let edges: BTreeSet = touching + .into_iter() + .filter(|(from, to, _)| { + (subject(from) || live.contains_key(from)) && (subject(to) || live.contains_key(to)) + }) + .map(|(from, to, kind)| nexus_flow_core::graph::Edge { from, to, kind }) + .collect(); + let mut rows = live; + for item in &listed { + if !held.contains(item.id.as_str()) { + rows.insert(item.id.clone(), item.clone()); + } + } + let ids: Vec<&str> = rows.keys().map(String::as_str).collect(); + let mut labels = if reads.labels { + store.labels_of_bulk(&ids)? + } else { + BTreeMap::new() + }; + let mut custom = if reads.custom { + store.custom_fields_of_bulk(&ids)? + } else { + BTreeMap::new() + }; + let mut graph = RankingGraph::new(reads.clone()); + graph.edges = edges; + graph.listed = listed_ids.iter().map(|id| id.to_string()).collect(); + graph.neighbours = rows + .into_iter() + .map(|(id, item)| { + let mut n = Neighbour::new(item); + n.labels = labels.remove(&id).unwrap_or_default(); + n.custom = custom.remove(&id).unwrap_or_default(); + (id, n) + }) + .collect(); + Ok(graph) +} + /// The store path's input as an [`ActiveBoard`]: the store's selection ([`graph::select`]) and, for /// every active ticket, its row and joins, each read in bulk. What [`next_active`] and the other /// store-free lanes answer from it is what [`next`] and its siblings answer from the store. @@ -2823,21 +3054,45 @@ pub fn active_board(store: &Store) -> Result { /// /// [`Engine::next`]: crate::engine::Engine::next /// -/// A `[ranking.next]` with derived ranking keys (6j6v.x0wf) is refused with `validation`: an -/// [`ActiveBoard`] holds neither the edges nor the scopes those keys read, and ranking without them -/// would order apart from the store path in silence. The server path is its own cut (6j6v.1c5s). +/// A `[ranking.next]` with derived ranking keys (6j6v.x0wf) is ranked from the board's +/// [`RankingGraph`] (6j6v.1c5s), by the same evaluation the store path runs — a key that reads the +/// rank of a type only `[ranking.list]` orders too, from the graph's listed items (6j6v.xkr7). It +/// is refused with `validation` when the board carries no graph, or one read for less than +/// [`next_ranking_reads`] names: ranking without those edges would order apart from the store path +/// in silence. pub fn next_active(cfg: &PluginConfig, board: &ActiveBoard, now: &str) -> Result> { crate::validate::iso_date(now)?; - if cfg.ranking.has_derived(Verb::Next) { - return Err(NxfError::validation(format!( - "plugin '{}' ranks `next` by derived ranking keys, which the store-free lanes do not \ - compute yet — answer `next` from a store", - cfg.name - ))); - } + let graph = match next_ranking_reads(cfg)? { + None => None, + Some(needed) => Some( + board + .ranking + .as_ref() + .filter(|g| g.reads.covers(&needed)) + .ok_or_else(|| { + NxfError::validation(format!( + "plugin '{}' ranks `next` by derived ranking keys, and the board was read \ + without the graph they read — fill `ActiveBoard::ranking` for \ + `next_ranking_reads` (`read::with_ranking` from a store, \ + `records::with_ranking` from the folded tables)", + cfg.name + )) + })?, + ), + }; let candidates = board.board.lanes(now).candidates; let items = board.items(candidates.iter().map(|c| &c.id)); - Ok(next_lane(cfg, candidates, items, DEFAULT_SORT_NEXT, None)) + let derived = match graph { + Some(graph) => derived::compute_active(cfg, board, graph, &items)?, + None => None, + }; + Ok(next_lane( + cfg, + candidates, + items, + DEFAULT_SORT_NEXT, + derived.as_ref(), + )) } /// [`next_active`], narrowed by `filter` (6j6v.15ed): the rows [`next_filtered`] gives over a store @@ -2929,6 +3184,401 @@ pub fn deferred_active_value(cfg: &PluginConfig, board: &ActiveBoard, now: &str) Ok(board.records(&items, |i, j| lane_record(Some(cfg), i, j))) } +// ---- list without a store (6j6v.xkr7) ---------------------------------------- +// +// The hosted `list`: the same rows in the same order as [`list_page`] over a store holding the same +// board, from a [`ListBoard`] a server reads off its `typed` index (docs/specs/derived-ranking-keys.md +// §6) — the listed types' live items with their joins, `next`'s board when a key reads a rank +// `next` gives, and the [`RankingGraph`] the derived keys read. As for the store-free `next`, the +// facade decides what to read ([`list_reads`]) and ranks and projects it; the server only reads. + +/// What the store-free `list` a [`ListQuery`] asks for reads (6j6v.xkr7) — the input of a reader +/// that builds the [`ListBoard`] ([`list_board`] from a store, `nxs_fold_ddb::records::list_board` +/// from the folded tables). [`list_reads`] gives it. +/// +/// `#[non_exhaustive]`: start from `ListReads::default()` and set the fields. +#[derive(Debug, Clone, Default, PartialEq, Eq)] +#[non_exhaustive] +pub struct ListReads { + /// The listed types: the board holds every live, unarchived item of each. `None`: every live, + /// unarchived item, whatever its type — a query without a type filter. + pub types: Option>, + /// A key reads the rank of a type `next` ranks: the board carries `next`'s board, the active + /// tickets and their [`Board`], to take `next`'s candidates from at the query's `now`. + pub next: bool, + /// The graph the derived keys read (`None`: none in play). Its subjects are the listed items, + /// the active tickets when `next` is set, and its own listed items + /// ([`RankingReads::list_types`]). + pub ranking: Option, + /// What a `list` that is NOT ranked keeps and where it cuts (review of PR #55): its order and + /// its row predicates read the item rows alone, so a reader keeps and cuts BEFORE it reads the + /// joins a record carries, and reads them for the kept page only ([`list_select`]). `None` for + /// `sort = rank`: a rank is computed over the whole `list` scope of a type — every live, + /// unarchived item of it, whatever the query's status, filter or limit + /// (docs/specs/derived-ranking-keys.md §12, decision 2) — so a ranked list reads the scope + /// whole. + pub select: Option, +} + +/// The query of a `list` that is not ranked, as a reader applies it before the joins +/// ([`ListReads::select`], [`list_select`]). +/// +/// `#[non_exhaustive]`: only [`list_reads`] builds one. +#[derive(Debug, Clone, PartialEq, Eq)] +#[non_exhaustive] +pub struct ListSelect { + /// Only items with this core status. + pub status: Option, + /// The query's filter: its types read the row, its label the item's labels, its container the + /// item's current parent and `contributes_to` targets. + pub filter: NextFilter, + /// The order — any key but `rank` ([`ListSelect::sort`]); it reads the row alone. + pub(crate) sort: SortKey, + /// The cut; `None`: every kept item. + pub limit: Option, +} + +impl ListSelect { + /// The order: any key but `rank`, which reads the row alone. + pub fn sort(&self) -> SortKey { + self.sort + } + + /// Whether an item passes what its row alone decides: live, unarchived, of the status and of a + /// type the filter names. A reader asks it first, and reads no join of an item it refuses. + pub fn keeps_row(&self, item: &ItemRow) -> bool { + item.deleted.as_deref() != Some("1") + && item.archived.is_none() + && self + .status + .as_deref() + .is_none_or(|s| item.status.as_deref() == Some(s)) + && (self.filter.types.is_empty() + || item + .item_type + .as_deref() + .is_some_and(|t| self.filter.types.iter().any(|f| f == t))) + } + + /// Whether the filter reads an item's labels: a reader reads them before [`list_select`]. + pub fn reads_labels(&self) -> bool { + self.filter.label.is_some() + } + + /// Whether the filter reads an item's current parent (`belongs_to`) and `contributes_to` + /// targets: a reader reads them before [`list_select`]. + pub fn reads_edges(&self) -> bool { + self.filter.within.is_some() + } +} + +/// Where a reader cut a [`ListBoard`] ([`list_select`]): the query it kept and cut for, and how many +/// items it kept before the cut — the total the page discloses. +/// +/// `#[non_exhaustive]`: only [`list_select`] builds one. +#[derive(Debug, Clone, PartialEq, Eq)] +#[non_exhaustive] +pub struct ListCut { + pub select: ListSelect, + pub total: usize, +} + +/// The items of `items` that `select` keeps — [`ListSelect::keeps_row`] and the filter's one +/// membership rule ([`NextFilter`]) — in `select.sort` order, cut to `select.limit`, with the +/// [`ListCut`] to put on the board ([`ListBoard::cut`]). What [`list_board_page`] keeps, orders and +/// cuts, so a reader that reads the joins of the kept items only answers the same page. +/// +/// Every item must carry its labels when [`ListSelect::reads_labels`], and its `belongs_to` and +/// `contributes_to` when [`ListSelect::reads_edges`]; no other join is read here. +pub fn list_select(select: &ListSelect, items: Vec) -> (Vec, ListCut) { + let mut kept: Vec = items + .into_iter() + .filter(|i| { + select.keeps_row(&i.item) && select.filter.keeps(&i.item, &i.labels, &i.contributes_to) + }) + .collect(); + // Ties by id: `list_board_page` sorts the board's items stably from id order, so ties are cut + // alike. + kept.sort_by(|a, b| { + row_cmp(&a.item, &b.item, select.sort).then_with(|| a.item.id.cmp(&b.item.id)) + }); + let total = kept.len(); + if let Some(n) = select.limit { + kept.truncate(n); + } + ( + kept, + ListCut { + select: select.clone(), + total, + }, + ) +} + +/// One listed item and the joins its `list` record and the filters read (6j6v.xkr7) — the +/// per-item input of the store-free `list`. Every field holds what the store path reads for the +/// same item: `item` with its `belongs_to` (`present_parent`), `labels` as `present_labels`, +/// `custom` every non-empty custom value, the timestamps with an unstamped `''` as `None`, +/// `contributes_to` the present targets sorted and distinct, and `parent_closed` what +/// [`parent_closed_reason`] gives. +/// +/// `#[non_exhaustive]`: build it with [`ListItem::new`] and set the fields. +#[derive(Debug, Clone, PartialEq, Eq)] +#[non_exhaustive] +pub struct ListItem { + pub item: ItemRow, + pub labels: Vec, + pub custom: BTreeMap, + pub created_at: Option, + pub updated_at: Option, + pub contributes_to: Vec, + pub parent_closed: Vec<(String, Option)>, +} + +impl ListItem { + /// An item with no joins yet. + pub fn new(item: ItemRow) -> ListItem { + ListItem { + item, + labels: Vec::new(), + custom: BTreeMap::new(), + created_at: None, + updated_at: None, + contributes_to: Vec::new(), + parent_closed: Vec::new(), + } + } + + fn joins(&self) -> RecordJoins<'_> { + RecordJoins { + labels: &self.labels, + parent_closed: &self.parent_closed, + created_at: self.created_at.as_deref(), + updated_at: self.updated_at.as_deref(), + custom: Some(&self.custom), + ..RecordJoins::default() + } + } +} + +/// The input of the store-free `list` (6j6v.xkr7): what a reader read for a [`ListReads`]. +/// +/// `#[non_exhaustive]`: build it with [`ListBoard::new`] and set the fields. +#[derive(Debug, Clone, Default)] +#[non_exhaustive] +pub struct ListBoard { + /// The types `items` holds every live, unarchived item of; `None`: every type. + pub types: Option>, + /// The listed items, by id: live and unarchived, of `types`. + pub items: BTreeMap, + /// `next`'s board, for a key that reads the rank of a type `next` ranks + /// ([`ListReads::next`]): the active tickets' rows — their labels and custom values when the + /// graph reads them — and their [`Board`]. Its own `ranking` stays `None`. + pub active: Option, + /// The graph the derived keys read ([`ListReads::ranking`]). + pub ranking: Option, + /// Set when the reader kept and cut `items` for a [`ListReads::select`] ([`list_select`]): + /// `items` then holds the kept page only, and the cut the total before it. `None`: `items` + /// holds every listed item. + pub cut: Option, +} + +impl ListBoard { + /// The listed items of `types` (`None`: every type), keyed by their ids. + pub fn new( + types: Option>, + items: impl IntoIterator, + ) -> ListBoard { + ListBoard { + types, + items: items.into_iter().map(|i| (i.item.id.clone(), i)).collect(), + active: None, + ranking: None, + cut: None, + } + } +} + +/// What the store-free `list` `q` reads under `cfg` (6j6v.xkr7): the listed types — `q`'s type +/// filter, else every type —, `next`'s board when a `[ranking.list]` key reads a rank `next` gives +/// and `q` has a `now`, and the graph the derived keys of a `sort = rank` read. It depends on the +/// query, not on the board. +pub fn list_reads(cfg: &PluginConfig, q: &ListQuery) -> ListReads { + derived::reads_for_list(cfg, q) +} + +/// The store's [`ListBoard`] for `reads` — the parity oracle for +/// `nxs_fold_ddb::records::list_board`, as [`active_board`] is for the records: the listed items +/// with their joins read in bulk, `next`'s board from the store's selection, and the graph. +pub fn list_board(store: &Store, reads: &ListReads) -> Result { + let items: Vec = derived::list_scope(store)? + .into_iter() + .filter(|i| { + reads + .types + .as_ref() + .is_none_or(|types| i.item_type.as_deref().is_some_and(|t| types.contains(t))) + }) + .collect(); + let ids: Vec<&str> = items.iter().map(|i| i.id.as_str()).collect(); + let mut labels = store.labels_of_bulk(&ids)?; + let mut custom = store.custom_fields_of_bulk(&ids)?; + let mut ts = store.item_timestamps_of_bulk(&ids)?; + let mut contributes = store.contributes_to_of_bulk(&ids)?; + let mut listed = Vec::with_capacity(items.len()); + for item in items { + let id = item.id.clone(); + let (created_at, updated_at) = ts.remove(&id).unwrap_or_default(); + let mut i = ListItem::new(item); + i.labels = labels.remove(&id).unwrap_or_default(); + i.custom = custom.remove(&id).unwrap_or_default(); + i.created_at = created_at; + i.updated_at = updated_at; + i.contributes_to = contributes.remove(&id).unwrap_or_default(); + listed.push(i); + } + // Kept and cut as a server keeps and cuts it (`list_select`), so the two boards compare whole. + let cut = match &reads.select { + Some(select) => { + let (kept, cut) = list_select(select, listed); + listed = kept; + Some(cut) + } + None => None, + }; + for i in &mut listed { + i.parent_closed = parent_closed_reason(store, &i.item)?; + } + let mut board = ListBoard::new(reads.types.clone(), listed); + board.cut = cut; + let graph_reads = reads.ranking.clone().unwrap_or_default(); + if reads.next { + let selection = nexus_flow_core::graph::select(store.connection())?; + let ids: Vec<&str> = selection.tickets().map(|t| t.id.as_str()).collect(); + let items = store.get_items(&ids)?; + let mut labels = if graph_reads.labels { + store.labels_of_bulk(&ids)? + } else { + BTreeMap::new() + }; + let mut custom = if graph_reads.custom { + store.custom_fields_of_bulk(&ids)? + } else { + BTreeMap::new() + }; + let tickets = items.into_iter().map(|item| { + let id = item.id.clone(); + let mut t = ActiveTicket::new(item); + t.labels = labels.remove(&id).unwrap_or_default(); + t.custom = custom.remove(&id).unwrap_or_default(); + t + }); + board.active = Some(ActiveBoard::new(selection, tickets)); + } + if let Some(r) = &reads.ranking { + let mut held: BTreeSet<&str> = board.items.keys().map(String::as_str).collect(); + if let Some(active) = &board.active { + held.extend(active.tickets.keys().map(String::as_str)); + } + let graph = ranking_graph(store, r, &held)?; + board.ranking = Some(graph); + } + Ok(board) +} + +/// [`list_page`] without a store (6j6v.xkr7): the rows of `board` that `q` keeps — its `status`, +/// its filter through the same membership rule ([`NextFilter`]) — ordered by `q.sort`, with the +/// derived keys of a `sort = rank` from the board's graph, and cut to `q.limit` with the total +/// disclosed. The same rows in the same order as [`list_page`] over a store holding the same board. +/// +/// Not paged: the store path's page cache is machine-local, and a server's paging is an open +/// decision (6j6v.15ed) — as for [`next_active_filtered`], a server cuts this list itself. So +/// `validation` for `q.paginate` or a token; and for a malformed `q.now`, a type filter (or none) +/// the board does not list every item for, a board read for less than the ranking needs, and a +/// board kept and cut ([`ListBoard::cut`]) for another query than `q`. A cut board answers the +/// total its reader counted before the cut. +pub fn list_board_page(cfg: &PluginConfig, board: &ListBoard, q: &ListQuery) -> Result { + if q.paginate || q.token.is_some() { + return Err(NxfError::validation( + "the store-free `list` is not paged — a server cuts it itself; drop paginate and the \ + token", + )); + } + if let Some(now) = &q.now { + crate::validate::iso_date(now)?; + } + let listed = match &board.types { + None => true, + Some(types) => { + !q.filter.types.is_empty() && q.filter.types.iter().all(|t| types.contains(t)) + } + }; + if !listed { + return Err(NxfError::validation( + "the list board was read for other types than the query lists — read it with \ + `list_reads` for this query", + )); + } + if let Some(cut) = &board.cut { + if list_reads(cfg, q).select.as_ref() != Some(&cut.select) { + return Err(NxfError::validation( + "the list board was kept and cut for another query — read it with `list_reads` \ + for this query", + )); + } + } + let mut items: Vec = board + .items + .values() + // The board holds live, unarchived items; a caller-built one is held to the same rule. + .filter(|i| i.item.deleted.as_deref() != Some("1") && i.item.archived.is_none()) + .filter(|i| { + q.status + .as_deref() + .is_none_or(|s| i.item.status.as_deref() == Some(s)) + }) + .filter(|i| q.filter.keeps(&i.item, &i.labels, &i.contributes_to)) + .map(|i| i.item.clone()) + .collect(); + let key = q.sort.unwrap_or(DEFAULT_SORT_LIST); + if key == SortKey::Rank { + let derived = derived::compute_list(cfg, board, &items, q.now.as_deref())?; + let rank = TypeRank::list(cfg).with_derived(derived.as_ref()); + items.sort_by(|a, b| rank.cmp(a, b)); + } else { + order_by(cfg, &mut items, key); + } + let mut page = truncate_next(items, q.limit); + if let Some(cut) = &board.cut { + // The reader kept the page only: the total is the one it counted before its cut. + page.total = cut.total; + } + Ok(page) +} + +/// [`list_board_page`] as the `list` JSON envelope `{"items", "total"}` — the value +/// [`Engine::list_query_value`] gives for the same board and an unpaged query, every record with +/// its joins from the item's [`ListItem`]. +/// +/// [`Engine::list_query_value`]: crate::engine::Engine::list_query_value +pub fn list_board_value(cfg: &PluginConfig, board: &ListBoard, q: &ListQuery) -> Result { + let page = list_board_page(cfg, board, q)?; + let records = Value::Array( + page.items + .iter() + .map(|i| { + let joins = board + .items + .get(&i.id) + .map(ListItem::joins) + .unwrap_or_default(); + list_record(Some(cfg), i, &joins) + }) + .collect(), + ); + Ok(next_page_envelope(records, &page, false)) +} + // ---- lane-ranked search (C6 #916.6) ---------------------------------------- // // `search` is a facade method (not CLI-only) so the CLI and an in-process consumer share ONE diff --git a/crates/facade/src/read/derived.rs b/crates/facade/src/read/derived.rs index 86e3e3a..0575049 100644 --- a/crates/facade/src/read/derived.rs +++ b/crates/facade/src/read/derived.rs @@ -19,10 +19,27 @@ //! //! So a query costs O(E + N log N) in the relevant edges and the items in scope, and a plugin //! that declares no derived key pays nothing: [`compute`] returns `None` before any read. +//! +//! **Without a store** (6j6v.1c5s). The store-free `next` ([`super::next_active`]) ranks a board a +//! server read: [`compute_active`] takes steps 1 and 2 from the board's [`RankingGraph`] — the +//! edges, already projected and filtered to live ends, and the live neighbours' rows, labels and +//! custom values — and runs steps 3 and 4 on the same [`Context`], so the two paths cannot evaluate +//! a key apart. A neighbour counts when it is LIVE, closed and archived ones included, on both +//! paths (spec §6, owner 2026-10-11). What the graph must hold is [`reads_for_next`]. +//! +//! The store-free `list` (6j6v.xkr7, [`super::list_board_page`]) ranks a [`ListBoard`] the same +//! way: [`compute_list`] takes the listed rows from the board, `next`'s candidates — when a key +//! reads a rank `next` gives — from its active board, and every other row from its graph. A +//! `[ranking.list]` rank a key reads is computed over the type's live items, which the graph holds +//! for every type in [`RankingReads::list_types`] (the `typed` index on a server), so a `next` key +//! may read one too ([`compute_active`]). What a `list` reads is [`reads_for_list`]. -use super::{lanes, mentioners_of, resolve_items}; +use super::{ + lanes, mentioners_of, resolve_items, ActiveBoard, ListBoard, ListQuery, ListReads, ListSelect, + RankingGraph, RankingReads, SortKey, DEFAULT_SORT_LIST, +}; use crate::error::{NxfError, Result}; -use crate::plugin::{Dir, Nulls, Pick, PluginConfig, RankKey, RankOf, Verb}; +use crate::plugin::{Dir, Nulls, Pick, PluginConfig, RankKey, RankOf, RankSpec, Verb}; use crate::record::{raw_field, RANKABLE_FIELDS}; use nexus_flow_core::model::{EdgeKind, ItemRow}; use nexus_flow_core::store::Store; @@ -131,20 +148,69 @@ pub(super) fn list_scope(store: &Store) -> Result> { .collect()) } -/// The values the derived keys of `verb`'s policy rank `ranked` by — `None`, before any read, -/// when the policy has no derived key. `validation` when a `list` key reads the rank of a type -/// `next` ranks and the caller passed neither `next`'s candidates nor a reference time. -pub(super) fn compute( - cfg: &PluginConfig, - store: &Store, - verb: Verb, - ranked: &[ItemRow], - scopes: Scopes<'_>, -) -> Result> { +/// A live row and its joins, wherever a store-free board holds it: an active ticket, a listed item, +/// or a neighbour in the graph. +#[derive(Clone, Copy)] +pub(super) struct Joined<'b> { + item: &'b ItemRow, + labels: &'b [String], + custom: &'b BTreeMap, +} + +/// The row `id` of an [`ActiveBoard`] with its `graph`: an active ticket's, else a neighbour's. +fn active_joined<'b>( + board: &'b ActiveBoard, + graph: &'b RankingGraph, + id: &str, +) -> Option> { + match board.tickets.get(id) { + Some(t) => Some(Joined { + item: &t.item, + labels: &t.labels, + custom: &t.custom, + }), + None => graph.neighbours.get(id).map(|n| Joined { + item: &n.item, + labels: &n.labels, + custom: &n.custom, + }), + } +} + +/// The row `id` of a [`ListBoard`] with its `graph`: a listed item's, else an active ticket's, +/// else a neighbour's. +fn list_joined<'b>(board: &'b ListBoard, graph: &'b RankingGraph, id: &str) -> Option> { + if let Some(i) = board.items.get(id) { + return Some(Joined { + item: &i.item, + labels: &i.labels, + custom: &i.custom, + }); + } + match &board.active { + Some(active) => active_joined(active, graph, id), + None => graph.neighbours.get(id).map(|n| Joined { + item: &n.item, + labels: &n.labels, + custom: &n.custom, + }), + } +} + +/// The derived side of one query, decided before any read: the type chain the ranked rows' orders +/// read the rank of (leaves first), and what the derived keys in play read. +struct Plan<'c> { + spec: &'c RankSpec, + chain: Vec, + needs: Needs, +} + +/// The [`Plan`] for ranking `ranked` under `verb` — `None` when the policy has no derived key. +fn plan<'c>(cfg: &'c PluginConfig, verb: Verb, ranked: &[ItemRow]) -> Option> { let ranking = &cfg.ranking; let spec = ranking.spec(verb); if !spec.has_derived() { - return Ok(None); + return None; } // The chain, leaves first: the orders the ranked rows' orders read the rank of, transitively. @@ -158,10 +224,128 @@ pub(super) fn compute( } } + // Every derived key in play. + let mut keys: Vec<&RankKey> = Vec::new(); + for ty in &ranked_types { + keys.extend(spec.order_for(*ty).iter().filter(|k| k.is_derived())); + } + for (v, ty) in &chain { + keys.extend( + ranking.spec(*v).by_type[ty] + .order + .iter() + .filter(|k| k.is_derived()), + ); + } + let needs = Needs::of(&keys); + Some(Plan { spec, chain, needs }) +} + +impl Plan<'_> { + /// The rows each order of the chain ranks: its type's rows in the scope of its verb. + fn node_rows<'r>(&self, next: &'r [ItemRow], list: &'r [ItemRow]) -> Vec> { + self.chain + .iter() + .map(|(v, ty)| { + let scope = match v { + Verb::Next => next, + Verb::List => list, + }; + scope + .iter() + .filter(|i| i.item_type.as_deref() == Some(ty.as_str())) + .collect() + }) + .collect() + } + + /// Whether some order of the chain is a `[ranking.list]` order. + fn reads_list(&self) -> bool { + self.chain.iter().any(|(v, _)| *v == Verb::List) + } + + /// Whether some order of the chain is a `[ranking.next]` order. + fn reads_next(&self) -> bool { + self.chain.iter().any(|(v, _)| *v == Verb::Next) + } + + /// What a [`RankingGraph`] must hold for this plan: its keys' reads, and the live items of + /// every `[ranking.list]` type of the chain the board does not list itself (`held`). + fn reads(&self, held: impl Fn(&str) -> bool) -> RankingReads { + let mut reads = self.needs.reads(); + reads.list_types = self + .chain + .iter() + .filter(|(v, ty)| *v == Verb::List && !held(ty)) + .map(|(_, ty)| ty.clone()) + .collect(); + reads + } +} + +/// The rows of a graph's listed items (the `list` scope of [`RankingReads::list_types`]), looked +/// up through `rows` — every one of them is a live row the board holds. +fn listed_rows<'b>( + graph: &RankingGraph, + rows: impl Fn(&str) -> Option>, +) -> Vec { + graph + .listed + .iter() + .filter_map(|id| rows(id).map(|j| j.item.clone())) + .collect() +} + +/// The graph a store-free query ranks with: the board's, when it was read for `needed`; an empty +/// one when the keys in play read nothing beyond the rows. `validation` otherwise — ranking without +/// those edges would order apart from the store path in silence. +fn graph_for<'g>( + graph: Option<&'g RankingGraph>, + needed: &RankingReads, + empty: &'g RankingGraph, + what: &str, +) -> Result<&'g RankingGraph> { + if *needed == RankingReads::default() { + return Ok(graph.unwrap_or(empty)); + } + graph.filter(|g| g.reads.covers(needed)).ok_or_else(|| { + NxfError::validation(format!( + "{what} ranks by derived ranking keys, and the board was read without the graph they \ + read — fill its `ranking` for the reads its plugin names" + )) + }) +} + +/// Every row a key is evaluated on: the ranked rows and the rows of the chain's scopes. +fn subjects<'a>( + ranked: &'a [ItemRow], + node_rows: &[Vec<&'a ItemRow>], +) -> HashMap<&'a str, &'a ItemRow> { + let mut subjects: HashMap<&str, &ItemRow> = HashMap::new(); + for row in ranked.iter().chain(node_rows.iter().flatten().copied()) { + subjects.insert(row.id.as_str(), row); + } + subjects +} + +/// The values the derived keys of `verb`'s policy rank `ranked` by — `None`, before any read, +/// when the policy has no derived key. `validation` when a `list` key reads the rank of a type +/// `next` ranks and the caller passed neither `next`'s candidates nor a reference time. +pub(super) fn compute( + cfg: &PluginConfig, + store: &Store, + verb: Verb, + ranked: &[ItemRow], + scopes: Scopes<'_>, +) -> Result> { + let Some(plan) = plan(cfg, verb, ranked) else { + return Ok(None); + }; + // The scopes the chain ranks over, read only when the caller does not hold them. let next_owned; let next_rows: &[ItemRow] = match scopes.next { - _ if !chain.iter().any(|(v, _)| *v == Verb::Next) => &[], + _ if !plan.chain.iter().any(|(v, _)| *v == Verb::Next) => &[], Some(rows) => rows, None => { let now = scopes.now.ok_or_else(|| { @@ -178,89 +362,220 @@ pub(super) fn compute( }; let list_owned; let list_rows: &[ItemRow] = match scopes.list { - _ if !chain.iter().any(|(v, _)| *v == Verb::List) => &[], + _ if !plan.reads_list() => &[], Some(rows) => rows, None => { list_owned = list_scope(store)?; &list_owned } }; - let node_rows: Vec> = chain - .iter() - .map(|(v, ty)| { - let scope = match v { - Verb::Next => next_rows, - Verb::List => list_rows, - }; - scope - .iter() - .filter(|i| i.item_type.as_deref() == Some(ty.as_str())) - .collect() - }) + let node_rows = plan.node_rows(next_rows, list_rows); + let mut ctx = Context::new(cfg, subjects(ranked, &node_rows)); + ctx.read_edges(store, &plan.needs)?; + Ok(Some(ctx.finish(&plan, verb, ranked, &node_rows))) +} + +/// [`compute`] for the store-free `next` (6j6v.1c5s): the values the derived keys of +/// `[ranking.next]` rank `candidates` — every candidate of the board — by, read from the +/// [`RankingGraph`] a reader filled instead of from a store. The candidates are `next`'s scope for +/// "rank", as on the store path; the `list` scope of a `[ranking.list]` type the chain reaches is +/// the graph's listed items (6j6v.xkr7). `None` when the policy has no derived key. +/// +/// The same [`Context`] evaluates the keys as on the store path; only where its edges, rows, +/// labels and custom values come from differs ([`Context::read_graph`]). The caller, +/// [`next_active`](super::next_active), has refused a graph read for less than +/// [`next_ranking_reads`](super::next_ranking_reads) names, which covers every plan of the section. +pub(super) fn compute_active( + cfg: &PluginConfig, + board: &ActiveBoard, + graph: &RankingGraph, + candidates: &[ItemRow], +) -> Result> { + let Some(plan) = plan(cfg, Verb::Next, candidates) else { + return Ok(None); + }; + // `next_active` admitted the graph only when it covers `next_ranking_reads`, which gathers the + // reads of every order of the section and of its chain — so of this plan's too. + debug_assert!(graph.reads.covers(&plan.reads(|_| false))); + let rows = |id: &str| active_joined(board, graph, id); + let listed = listed_rows(graph, rows); + let node_rows = plan.node_rows(candidates, &listed); + let mut ctx = Context::new(cfg, subjects(candidates, &node_rows)); + ctx.read_graph(rows, graph, &plan.needs); + Ok(Some(ctx.finish(&plan, Verb::Next, candidates, &node_rows))) +} + +/// [`compute`] for the store-free `list` (6j6v.xkr7): the values the derived keys of +/// `[ranking.list]` rank `ranked` by, from a [`ListBoard`]. The `list` scope of a type is its live, +/// unarchived items — the board's listed items for the types it lists, the graph's listed items +/// for the others; `next`'s scope, when the chain reaches a `[ranking.next]` order, is the +/// candidates of the board's active board at `now`. `None` when the policy has no derived key. +/// +/// `validation` as on the store path when the chain reads a `next` rank and `now` is missing, and +/// when the board was read for less than the plan needs: no active board for such a chain, a +/// graph read for fewer reads, a chain type neither listed nor in the graph. +pub(super) fn compute_list( + cfg: &PluginConfig, + board: &ListBoard, + ranked: &[ItemRow], + now: Option<&str>, +) -> Result> { + let Some(plan) = plan(cfg, Verb::List, ranked) else { + return Ok(None); + }; + let held = |ty: &str| board.types.as_ref().is_none_or(|t| t.contains(ty)); + let needed = plan.reads(held); + let empty = RankingGraph::default(); + let graph = graph_for( + board.ranking.as_ref(), + &needed, + &empty, + &format!("plugin '{}' (`list`)", cfg.name), + )?; + let next_owned; + let next_rows: &[ItemRow] = if plan.reads_next() { + let now = now.ok_or_else(|| { + NxfError::validation( + "the plugin's [ranking.list] reads the rank of a type that `next` ranks, which is \ + computed over `next`'s candidates at a reference time — pass `now`", + ) + })?; + let active = board.active.as_ref().ok_or_else(|| { + NxfError::validation(format!( + "plugin '{}' ranks `list` by the rank of a type that `next` ranks, and the board \ + was read without the active board that rank is computed over — read it with \ + `ListReads::next`", + cfg.name + )) + })?; + let candidates = active.board.lanes(now).candidates; + next_owned = active.items(candidates.iter().map(|c| &c.id)); + &next_owned + } else { + &[] + }; + let rows = |id: &str| list_joined(board, graph, id); + let list_owned: Vec = if plan.reads_list() { + let mut scope: Vec = board.items.values().map(|i| i.item.clone()).collect(); + scope.extend( + listed_rows(graph, rows) + .into_iter() + .filter(|i| !board.items.contains_key(&i.id)), + ); + scope + } else { + Vec::new() + }; + let node_rows = plan.node_rows(next_rows, &list_owned); + let mut ctx = Context::new(cfg, subjects(ranked, &node_rows)); + ctx.read_graph(rows, graph, &plan.needs); + Ok(Some(ctx.finish(&plan, Verb::List, ranked, &node_rows))) +} + +/// What the store-free `next` of `cfg` reads beyond the active tickets' records (6j6v.1c5s): +/// `None` when `[ranking.next]` declares no derived key. Gathered over every order of the section +/// and of its type chain, so it does not depend on which types a board holds. Since 6j6v.xkr7 an +/// order that reads the rank of a type only `[ranking.list]` orders is no longer refused: the +/// type lands in [`RankingReads::list_types`], whose live items a reader puts in the graph. +pub(super) fn reads_for_next(cfg: &PluginConfig) -> Result> { + let spec = cfg.ranking.spec(Verb::Next); + if !spec.has_derived() { + return Ok(None); + } + let orders: Vec<&[RankKey]> = std::iter::once(spec.order.as_slice()) + .chain(spec.by_type.values().map(|o| o.order.as_slice())) .collect(); + // `next`'s own per-type orders start the chain: an order there may read another's rank. + let starts = spec.by_type.keys().map(|ty| (Verb::Next, ty.clone())); + Ok(Some(chain_reads(cfg, &orders, starts, |_| false).0)) +} - // Every row a key is evaluated on, and every derived key in play. - let mut subjects: HashMap<&str, &ItemRow> = HashMap::new(); - for row in ranked.iter().chain(node_rows.iter().flatten().copied()) { - subjects.insert(row.id.as_str(), row); +/// What the store-free `list` `q` asks for reads (6j6v.xkr7): its listed types (`None`: every +/// type), whether `next`'s board is needed, and the graph. Only `sort = rank` ranks by the +/// derived keys; any other order reads the listed items alone. +pub(super) fn reads_for_list(cfg: &PluginConfig, q: &ListQuery) -> ListReads { + let types: Option> = + (!q.filter.types.is_empty()).then(|| q.filter.types.iter().cloned().collect()); + let mut reads = ListReads { + types: types.clone(), + next: false, + ranking: None, + select: None, + }; + // Any order but `rank` reads the item rows alone: the reader may keep and cut before it reads + // an item's joins (review of PR #55, Integrity #1). + let sort = q.sort.unwrap_or(DEFAULT_SORT_LIST); + if sort != SortKey::Rank { + reads.select = Some(ListSelect { + status: q.status.clone(), + filter: q.filter.clone(), + sort, + limit: q.limit, + }); + return reads; } - let mut keys: Vec<&RankKey> = Vec::new(); - for ty in &ranked_types { - keys.extend(spec.order_for(*ty).iter().filter(|k| k.is_derived())); + let spec = cfg.ranking.spec(Verb::List); + if !spec.has_derived() { + return reads; } + let orders: Vec<&[RankKey]> = match &types { + Some(types) => types + .iter() + .map(|ty| spec.order_for(Some(ty.as_str()))) + .collect(), + None => std::iter::once(spec.order.as_slice()) + .chain(spec.by_type.values().map(|o| o.order.as_slice())) + .collect(), + }; + let starts: Vec = orders + .iter() + .flat_map(|keys| cfg.ranking.rank_deps(keys, Verb::List)) + .collect(); + let held = |ty: &str| types.as_ref().is_none_or(|t| t.contains(ty)); + let (graph, reads_next) = chain_reads(cfg, &orders, starts, held); + // Without `now` a chain that reads a `next` rank answers `validation` where a ranked row + // needs it, as on the store path — reading `next`'s board for it would be wasted. + reads.next = reads_next && q.now.is_some(); + if graph != RankingReads::default() { + reads.ranking = Some(graph); + } + reads +} + +/// The reads of `orders` and of every order down the type chain from `starts`, the chain's +/// `[ranking.list]` types not `held` among them, and whether that chain holds a `[ranking.next]` +/// order. +fn chain_reads( + cfg: &PluginConfig, + orders: &[&[RankKey]], + starts: impl IntoIterator, + held: impl Fn(&str) -> bool, +) -> (RankingReads, bool) { + let mut chain: Vec = Vec::new(); + let mut seen: BTreeSet = BTreeSet::new(); + for node in starts { + post_order(cfg, node, &mut seen, &mut chain); + } + let mut keys: Vec<&RankKey> = orders + .iter() + .flat_map(|o| o.iter()) + .filter(|k| k.is_derived()) + .collect(); for (v, ty) in &chain { keys.extend( - ranking.spec(*v).by_type[ty] + cfg.ranking.spec(*v).by_type[ty] .order .iter() .filter(|k| k.is_derived()), ); } - - let needs = Needs::of(&keys); - let mut ctx = Context { - cfg, - subjects, - out: Adjacency::new(), - inb: Adjacency::new(), - mentioners: BTreeMap::new(), - neighbours: HashMap::new(), - labels: BTreeMap::new(), - custom: BTreeMap::new(), - ranks: HashMap::new(), - }; - ctx.read_edges(store, &needs)?; - - // Bottom-up: each order of the chain ranks its scope once; its dependents look the rank up. - for (node, rows) in chain.iter().zip(&node_rows) { - let keys = &ranking.spec(node.0).by_type[&node.1].order; - let values: Vec> = rows.iter().map(|r| ctx.values(keys, node.0, r)).collect(); - let mut order: Vec = (0..rows.len()).collect(); - let cmp = |a: usize, b: usize| { - super::keys_cmp_values(rows[a], rows[b], keys, Some(&values[a]), Some(&values[b])) - }; - order.sort_by(|a, b| cmp(*a, *b).then_with(|| rows[*a].id.cmp(&rows[*b].id))); - // A DENSE rank over the declared keys (spec §4): rows equal on every key share a rank, and - // the next distinct row takes the next integer. The id orders the sort, never the rank. - let mut ranks: HashMap = HashMap::with_capacity(rows.len()); - let mut rank = 0_i64; - for (n, &i) in order.iter().enumerate() { - if n > 0 && cmp(order[n - 1], i) != Ordering::Equal { - rank += 1; - } - ranks.insert(rows[i].id.clone(), rank); - } - ctx.ranks.insert(node.clone(), ranks); - } - - let values = ranked + let mut reads = Needs::of(&keys).reads(); + reads.list_types = chain .iter() - .map(|r| { - let keys = spec.order_for(r.item_type.as_deref()); - (r.id.clone(), ctx.values(keys, verb, r)) - }) + .filter(|(v, ty)| *v == Verb::List && !held(ty)) + .map(|(_, ty)| ty.clone()) .collect(); - Ok(Some(Derived { values })) + (reads, chain.iter().any(|(v, _)| *v == Verb::Next)) } /// Depth-first post-order over the type chain: every order a node reads the rank of lands before @@ -319,6 +634,21 @@ impl Needs { } } +impl Needs { + /// These needs as the [`RankingReads`] of a graph: `mentions` among the kinds when a count reads + /// `mentioned_by`. No list types; the caller adds them. + fn reads(&self) -> RankingReads { + let mut reads = RankingReads::default(); + reads.kinds.extend(self.kinds.iter().copied()); + if self.mentioned_by { + reads.kinds.insert(EdgeKind::Mentions); + } + reads.labels = self.labels; + reads.custom = self.custom; + reads + } +} + /// One count, from a `count` key or a `sum` summand. struct Count<'a> { kind: &'a str, @@ -381,7 +711,139 @@ struct Context<'a> { ranks: HashMap>, } -impl Context<'_> { +impl<'a> Context<'a> { + /// A context over `subjects`, nothing read yet. + fn new(cfg: &'a PluginConfig, subjects: HashMap<&'a str, &'a ItemRow>) -> Context<'a> { + Context { + cfg, + subjects, + out: Adjacency::new(), + inb: Adjacency::new(), + mentioners: BTreeMap::new(), + neighbours: HashMap::new(), + labels: BTreeMap::new(), + custom: BTreeMap::new(), + ranks: HashMap::new(), + } + } + + /// Rank the chain bottom-up — each order its scope once, its dependents look the rank up — and + /// then the ranked rows' values. + fn finish( + mut self, + plan: &Plan<'_>, + verb: Verb, + ranked: &[ItemRow], + node_rows: &[Vec<&ItemRow>], + ) -> Derived { + let ranking = &self.cfg.ranking; + for (node, rows) in plan.chain.iter().zip(node_rows) { + let keys = &ranking.spec(node.0).by_type[&node.1].order; + let values: Vec> = + rows.iter().map(|r| self.values(keys, node.0, r)).collect(); + let mut order: Vec = (0..rows.len()).collect(); + let cmp = |a: usize, b: usize| { + super::keys_cmp_values(rows[a], rows[b], keys, Some(&values[a]), Some(&values[b])) + }; + order.sort_by(|a, b| cmp(*a, *b).then_with(|| rows[*a].id.cmp(&rows[*b].id))); + // A DENSE rank over the declared keys (spec §4): rows equal on every key share a rank, + // and the next distinct row takes the next integer. The id orders the sort, never the + // rank. + let mut ranks: HashMap = HashMap::with_capacity(rows.len()); + let mut rank = 0_i64; + for (n, &i) in order.iter().enumerate() { + if n > 0 && cmp(order[n - 1], i) != Ordering::Equal { + rank += 1; + } + ranks.insert(rows[i].id.clone(), rank); + } + self.ranks.insert(node.clone(), ranks); + } + let values = ranked + .iter() + .map(|r| { + let keys = plan.spec.order_for(r.item_type.as_deref()); + (r.id.clone(), self.values(keys, verb, r)) + }) + .collect(); + Derived { values } + } + + /// The reads from a [`RankingGraph`] instead of a store (6j6v.1c5s): the same adjacency, reverse + /// mentions, neighbour rows, labels and custom values [`read_edges`](Self::read_edges) reads, + /// taken from the graph's edges and, through `rows`, from the board's rows — its active + /// tickets, its listed items — and the graph's neighbours. The graph holds every edge of the + /// kinds read that touches a subject with both ends live, `parent` as the `present_parent` + /// projection — the edges the store's pass yields for the same subjects. + fn read_graph<'b>( + &mut self, + rows: impl Fn(&str) -> Option>, + graph: &RankingGraph, + needs: &Needs, + ) { + let mut mention_sources: BTreeMap> = BTreeMap::new(); + for e in &graph.edges { + if needs.kinds.contains(&e.kind) { + if self.subjects.contains_key(e.to.as_str()) { + self.inb + .entry((e.kind, e.to.clone())) + .or_default() + .push(e.from.clone()); + } + if self.subjects.contains_key(e.from.as_str()) { + self.out + .entry((e.kind, e.from.clone())) + .or_default() + .push(e.to.clone()); + } + } + if needs.mentioned_by + && e.kind == EdgeKind::Mentions + && self.subjects.contains_key(e.to.as_str()) + { + mention_sources + .entry(e.to.clone()) + .or_default() + .insert(e.from.as_str()); + } + } + for (target, froms) in mention_sources { + let found: Vec = froms + .into_iter() + .filter_map(|f| rows(f).map(|j| j.item.clone())) + .collect(); + if !found.is_empty() { + self.mentioners.insert(target, found); + } + } + let missing: BTreeSet<&str> = self + .out + .values() + .chain(self.inb.values()) + .flatten() + .map(String::as_str) + .filter(|id| !self.subjects.contains_key(id)) + .collect(); + self.neighbours = missing + .into_iter() + .filter_map(|id| rows(id).map(|j| (id.to_string(), j.item.clone()))) + .collect(); + if needs.labels || needs.custom { + let mut ids: BTreeSet<&str> = self.neighbours.keys().map(String::as_str).collect(); + ids.extend(self.subjects.keys().copied()); + ids.extend(self.mentioners.values().flatten().map(|i| i.id.as_str())); + for id in ids { + let Some(j) = rows(id) else { continue }; + if needs.labels && !j.labels.is_empty() { + self.labels.insert(id.to_string(), j.labels.to_vec()); + } + if needs.custom && !j.custom.is_empty() { + self.custom.insert(id.to_string(), j.custom.clone()); + } + } + } + } + /// The reads: the edge pass, the reverse mentions, the neighbours' rows, labels and custom /// fields — each ONE bulk read (per 500 ids where the store chunks), whatever the row count. fn read_edges(&mut self, store: &Store, needs: &Needs) -> Result<()> { diff --git a/crates/facade/tests/derived_ranking.rs b/crates/facade/tests/derived_ranking.rs index ae831c4..d5ad251 100644 --- a/crates/facade/tests/derived_ranking.rs +++ b/crates/facade/tests/derived_ranking.rs @@ -11,6 +11,8 @@ //! - Determinism (§9): another edge insertion order ranks alike, and ties fall to the id. //! - Bounded cost (§9, a query-shape guard in the style of xn8s): the edge pass runs once per query //! however many candidates there are. +//! - Without a store (6j6v.1c5s, 6j6v.xkr7): `read::next_active` and `read::list_board_page` over +//! the boards a server reads rank as the store path does. //! //! Its own test binary, so the fixture plugin does not pollute the facade lib tests' "exactly two //! OSS plugins" assertion. @@ -359,8 +361,37 @@ fn the_rank_follows_the_board_on_the_next_query() { } #[test] -fn the_store_free_lanes_refuse_a_plugin_with_derived_next_keys() { - // The server path is its own cut (1c5s); ranking without the edges would differ in silence. +fn the_store_free_next_ranks_from_the_ranking_graph_as_the_store_does() { + // 6j6v.1c5s: the board a server reads carries the graph the keys read, and the store-free + // `next` ranks from it exactly as the store path does — the spec's three-type chain. + let tmp = workspace("derived-rank-fixture"); + spec_board(tmp.path()); + let cfg = plugin::load("derived-rank-fixture").unwrap(); + let s = writer(tmp.path()); + let reads = read::next_ranking_reads(&cfg) + .unwrap() + .expect("derived next keys"); + assert_eq!( + reads.kinds.iter().copied().collect::>(), + [EdgeKind::ContributesTo, EdgeKind::Parent], + "the edges the next orders follow or count" + ); + assert!(reads.custom && !reads.labels, "{reads:?}"); + let board = read::with_ranking(&s, read::active_board(&s).unwrap(), &reads).unwrap(); + let got = read::next_active(&cfg, &board, NOW).unwrap(); + assert_eq!(ids(&got), ids(&read::next(&cfg, &s, NOW, None).unwrap())); + assert_eq!(of_type(&got, "project"), PROJECTS); + assert_eq!(of_type(&got, "action"), ACTIONS); + assert_eq!( + read::next_active_value(&cfg, &board, NOW).unwrap(), + read::next_to_value_with_custom(&cfg, &s, &read::next(&cfg, &s, NOW, None).unwrap()) + .unwrap() + ); +} + +#[test] +fn the_store_free_next_refuses_a_board_read_without_the_graph_its_keys_read() { + // Ranking without the edges would order apart from the store path in silence. let tmp = workspace("derived-rank-fixture"); spec_board(tmp.path()); let cfg = plugin::load("derived-rank-fixture").unwrap(); @@ -369,6 +400,350 @@ fn the_store_free_lanes_refuse_a_plugin_with_derived_next_keys() { let err = read::next_active(&cfg, &board, NOW).unwrap_err(); assert_eq!(err.kind, ErrorKind::Validation); assert!(err.msg.contains("derived ranking keys"), "{}", err.msg); + // A graph read for less than the keys need is refused too: here without the custom values + // the goals' `order` lives in. + let mut reads = read::next_ranking_reads(&cfg).unwrap().unwrap(); + reads.custom = false; + let short = read::with_ranking(&s, read::active_board(&s).unwrap(), &reads).unwrap(); + let err = read::next_active(&cfg, &short, NOW).unwrap_err(); + assert_eq!(err.kind, ErrorKind::Validation); + // `blocked` and `deferred` rank by no derived key and need no graph. + read::blocked_active(&cfg, &board); + read::deferred_active(&cfg, &board, NOW).unwrap(); +} + +/// A plugin ranking `next` by its neighbours with no status filter, so closed, archived and +/// deleted neighbours each decide something (owner, 2026-10-11: a key reads every LIVE neighbour). +const LIVE_ORDERS: &str = r#" +name = "live-neighbours-fixture" +[description] +en = "x" +de = "y" +[priority] +labels = ["P0", "P1", "P2", "P3"] +[types] +list = ["goal", "project", "action"] +[vocabulary.status] +open = "open" +in_progress = "in progress" +closed = "closed" +[ranking.next] +types = ["project", "action"] +order = [ { field = "priority", dir = "asc" } ] +[ranking.next.project] +order = [ + { related = "contributes_to", type = "goal", of = { field = "order" }, pick = "min", nulls = "last" }, + { count = "children", dir = "desc" }, + { count = "mentioned_by", dir = "desc" }, +] +[ranking.next.action] +order = [ { related = "parent", type = "project", of = "rank", nulls = "last" } ] +[presentation.list] +columns = ["id"] +[fields.order] +type = "text" +on = ["goal"] +"#; + +#[test] +fn a_closed_archived_or_deleted_neighbour_reads_alike_with_and_without_a_store() { + // Goals: `gc` closed (order 1), `ga` archived (order 2), `go` open (order 3), `gd` deleted + // (order 0). Projects `pa`..`pd` contribute to one each: a closed or archived goal still gives + // its order, a deleted one gives none (null, last). Projects `q1`..`q4` have no goal: `q1` has + // three children — two closed, one archived — `q2` one open child, `q3` is mentioned by a + // closed project and `q4` by nothing. Every child and mentioner but `q2`'s is out of `next`. + let cfg: plugin::PluginConfig = toml::from_str(LIVE_ORDERS).unwrap(); + let mut s = Store::open_in_memory(1); + for (g, order) in [ + ("zz12.gc", "1"), + ("zz12.ga", "2"), + ("zz12.go", "3"), + ("zz12.gd", "0"), + ] { + item(&mut s, g, "goal", "1"); + custom(&mut s, g, "order", order); + } + close(&mut s, "zz12.gc"); + s.set_field("zz12.ga", "archived", Some("2026-06-01".into()), "t"); + for (p, g) in [ + ("zz12.pa", "zz12.gc"), + ("zz12.pb", "zz12.ga"), + ("zz12.pc", "zz12.gd"), + ("zz12.pd", "zz12.go"), + ] { + item(&mut s, p, "project", "1"); + edge(&mut s, p, g, EdgeKind::ContributesTo); + } + s.delete_item("zz12.gd", "t"); + for q in ["zz12.q1", "zz12.q2", "zz12.q3", "zz12.q4"] { + item(&mut s, q, "project", "1"); + } + for (a, parent) in [ + ("zz12.a1", "zz12.q1"), + ("zz12.a2", "zz12.q1"), + ("zz12.a3", "zz12.q1"), + ("zz12.a4", "zz12.q2"), + ] { + item(&mut s, a, "action", "1"); + s.set_parent(a, parent, "t").unwrap(); + } + close(&mut s, "zz12.a1"); + close(&mut s, "zz12.a2"); + s.set_field("zz12.a3", "archived", Some("2026-06-01".into()), "t"); + item(&mut s, "zz12.px", "project", "1"); + edge(&mut s, "zz12.px", "zz12.q3", EdgeKind::Mentions); + close(&mut s, "zz12.px"); + + let want = [ + "zz12.pa", "zz12.pb", "zz12.pd", "zz12.q1", "zz12.q2", "zz12.q3", "zz12.pc", "zz12.q4", + ]; + let store = read::next(&cfg, &s, NOW, None).unwrap(); + assert_eq!(of_type(&store, "project"), want, "the store path"); + let reads = read::next_ranking_reads(&cfg).unwrap().unwrap(); + let board = read::with_ranking(&s, read::active_board(&s).unwrap(), &reads).unwrap(); + let graph = board.ranking.as_ref().unwrap(); + for id in ["zz12.gc", "zz12.ga", "zz12.a1", "zz12.a3", "zz12.px"] { + assert!( + graph.neighbours.contains_key(id), + "{id} is a live neighbour" + ); + } + assert!( + !graph.neighbours.contains_key("zz12.gd"), + "a deleted one is not" + ); + assert!( + graph.edges.iter().all(|e| e.to != "zz12.gd"), + "nor an edge to it" + ); + assert_eq!( + ids(&read::next_active(&cfg, &board, NOW).unwrap()), + ids(&store) + ); +} + +#[test] +fn a_next_order_reading_the_rank_of_a_type_only_list_orders_ranks_from_the_listed_items() { + // Spec §12 point 1 lets a `next` key read the rank `[ranking.list]` gives a type. That rank is + // computed over the type's live, unarchived items, which a board of active tickets does not + // hold: since 6j6v.xkr7 the reads name the type (`list_types`) and the graph holds its items — + // a server reads them off its `typed` index — so the store-free `next` ranks as the store + // path does. A board read without them is refused. + let cfg: plugin::PluginConfig = toml::from_str( + r#" + name = "list-rank-fixture" + [description] + en = "x" + de = "y" + [priority] + labels = ["P0", "P1", "P2"] + [types] + list = ["action", "person"] + [vocabulary.status] + open = "open" + in_progress = "in progress" + closed = "closed" + [ranking.next] + order = [ { field = "priority" } ] + [ranking.next.action] + order = [ { related = "mentions", type = "person", of = "rank", nulls = "last" } ] + [ranking.list.person] + order = [ { count = "mentioned_by", dir = "desc" }, { field = "priority", dir = "asc" } ] + [presentation.list] + columns = ["id"] + "#, + ) + .unwrap(); + let mut s = Store::open_in_memory(1); + // Persons: u1 mentioned twice (rank 0); u2 and the closed u3 once each, u3 with the better + // priority; u4 archived (no rank: outside the list scope); u5 deleted. + for (u, prio) in [ + ("u1", "2"), + ("u2", "1"), + ("u3", "0"), + ("u4", "0"), + ("u5", "0"), + ] { + item(&mut s, &format!("zz12.{u}"), "person", prio); + } + close(&mut s, "zz12.u3"); + s.set_field("zz12.u4", "archived", Some("2026-06-01".into()), "t"); + for (a, person) in [ + ("zz12.a1", "zz12.u2"), + ("zz12.a2", "zz12.u1"), + ("zz12.a3", "zz12.u3"), + ("zz12.a4", "zz12.u4"), + ("zz12.a5", "zz12.u5"), + ] { + item(&mut s, a, "action", "1"); + edge(&mut s, a, person, EdgeKind::Mentions); + } + item(&mut s, "zz12.a6", "action", "1"); + edge(&mut s, "zz12.a6", "zz12.u1", EdgeKind::Mentions); + s.delete_item("zz12.u5", "t"); + + let reads = read::next_ranking_reads(&cfg) + .unwrap() + .expect("derived next keys"); + assert_eq!( + reads.list_types.iter().collect::>(), + ["person"], + "{reads:?}" + ); + let board = read::with_ranking(&s, read::active_board(&s).unwrap(), &reads).unwrap(); + let graph = board.ranking.as_ref().unwrap(); + assert_eq!( + graph.listed.iter().collect::>(), + ["zz12.u1", "zz12.u2", "zz12.u3"], + "the live, unarchived persons" + ); + let store = read::next(&cfg, &s, NOW, None).unwrap(); + assert_eq!( + of_type(&store, "action"), + ["zz12.a2", "zz12.a6", "zz12.a3", "zz12.a1", "zz12.a4", "zz12.a5"] + ); + assert_eq!( + ids(&read::next_active(&cfg, &board, NOW).unwrap()), + ids(&store) + ); + + // Without the person's items the graph is short of what the keys read: refused. + let mut short = reads.clone(); + short.list_types.clear(); + let board = read::with_ranking(&s, read::active_board(&s).unwrap(), &short).unwrap(); + assert_eq!( + read::next_active(&cfg, &board, NOW).unwrap_err().kind, + ErrorKind::Validation + ); +} + +#[test] +fn the_store_free_list_ranks_persons_as_list_query_does() { + // 6j6v.xkr7: the hosted `list` — `read::list_board_page` over a `ListBoard` read for + // `read::list_reads` — gives the rows, the order, the total and the JSON `Engine::list_query` + // gives: persons by a sum of mentions, then by the rank `next` gives the projects they + // contribute to (`now` takes `next`'s candidates), filtered by type, status, container and + // label as locally. + let tmp = workspace("derived-rank-fixture"); + spec_board(tmp.path()); + let cfg = plugin::load("derived-rank-fixture").unwrap(); + let s = writer(tmp.path()); + let engine = engine(&tmp); + let queries = [ + ListQuery::new() + .sort(SortKey::Rank) + .filter(NextFilter::new().with_type("person")) + .now(NOW), + ListQuery::new() + .sort(SortKey::Rank) + .filter(NextFilter::new().with_type("person")) + .now(NOW) + .limit(2), + ListQuery::new().sort(SortKey::Rank).now(NOW), + ListQuery::new() + .sort(SortKey::Rank) + .filter( + NextFilter::new() + .with_type("action") + .with_container("zz12.p5"), + ) + .now(NOW), + ListQuery::new() + .sort(SortKey::Rank) + .filter(NextFilter::new().with_label("waiting")) + .now(NOW), + ListQuery::new().status("closed"), + ListQuery::new() + .filter(NextFilter::new().with_type("project").with_type("goal")) + .sort(SortKey::Rank), + ]; + for q in &queries { + let reads = read::list_reads(&cfg, q); + let board = read::list_board(&s, &reads).unwrap(); + let page = read::list_board_page(&cfg, &board, q).unwrap(); + let want = engine.list_query(q).unwrap(); + assert_eq!(ids(&page.items), ids(&want.items), "{q:?}"); + assert_eq!(page.total, want.total, "{q:?}"); + assert_eq!( + read::list_board_value(&cfg, &board, q).unwrap(), + engine.list_query_value(q).unwrap(), + "{q:?}" + ); + } + let persons = read::list_board_page( + &cfg, + &read::list_board(&s, &read::list_reads(&cfg, &queries[0])).unwrap(), + &queries[0], + ) + .unwrap(); + assert_eq!(ids(&persons.items), PERSONS); + let reads = read::list_reads(&cfg, &queries[0]); + assert!( + reads.next, + "persons read the rank `next` gives projects: {reads:?}" + ); + assert_eq!( + reads.types, + Some(["person".to_string()].into_iter().collect()) + ); +} + +#[test] +fn the_store_free_list_refuses_what_its_board_was_not_read_for() { + let tmp = workspace("derived-rank-fixture"); + spec_board(tmp.path()); + let cfg = plugin::load("derived-rank-fixture").unwrap(); + let s = writer(tmp.path()); + let persons = ListQuery::new() + .sort(SortKey::Rank) + .filter(NextFilter::new().with_type("person")) + .now(NOW); + let board = read::list_board(&s, &read::list_reads(&cfg, &persons)).unwrap(); + let refused = |q: &ListQuery, board: &read::ListBoard| { + let err = read::list_board_page(&cfg, board, q).unwrap_err(); + assert_eq!(err.kind, ErrorKind::Validation, "{q:?}"); + err.msg + }; + // Not paged: a server cuts the list itself. + refused(&persons.clone().limit(2).paginate(), &board); + // Other types, or every type, than the board lists. + refused( + &ListQuery::new().filter(NextFilter::new().with_type("project")), + &board, + ); + refused(&ListQuery::new(), &board); + // Without `now`, `next`'s candidates cannot be taken — as on the store path. + let no_now = ListQuery::new() + .sort(SortKey::Rank) + .filter(NextFilter::new().with_type("person")); + let reads = read::list_reads(&cfg, &no_now); + assert!(!reads.next, "nothing to read `next`'s board for: {reads:?}"); + let msg = refused(&no_now, &read::list_board(&s, &reads).unwrap()); + assert!(msg.contains("pass `now`"), "{msg}"); + // A board without the graph its keys read. + let mut short = read::list_reads(&cfg, &persons); + short.ranking = None; + let msg = refused(&persons, &read::list_board(&s, &short).unwrap()); + assert!(msg.contains("derived ranking keys"), "{msg}"); + // Without `next`'s board. + let mut short = read::list_reads(&cfg, &persons); + short.next = false; + let msg = refused(&persons, &read::list_board(&s, &short).unwrap()); + assert!(msg.contains("active board"), "{msg}"); + // A list that is not ranked is kept and cut before its joins: its board holds the page of ONE + // query, and answers no other — not another status, not another limit (review of PR #55). + let closed = ListQuery::new().status("closed").limit(1); + let cut = read::list_board(&s, &read::list_reads(&cfg, &closed)).unwrap(); + assert!(cut.cut.is_some() && cut.items.len() <= 1); + read::list_board_page(&cfg, &cut, &closed).unwrap(); + for other in [ + ListQuery::new().status("open").limit(1), + ListQuery::new().status("closed").limit(2), + ListQuery::new().status("closed"), + ] { + let msg = refused(&other, &cut); + assert!(msg.contains("kept and cut for another query"), "{msg}"); + } } // ---- each kind over hand-built boards (§9 "per kind") ------------------------------------- diff --git a/crates/fold-ddb/Cargo.toml b/crates/fold-ddb/Cargo.toml index 97c413c..d1d1e7b 100644 --- a/crates/fold-ddb/Cargo.toml +++ b/crates/fold-ddb/Cargo.toml @@ -2,7 +2,7 @@ name = "nxs-fold-ddb" version.workspace = true edition = "2021" -description = "Folding on a server (6j6v.vvw6): the reducers' described changes carried out against DynamoDB, the board's active and dated indexes, next/blocked over them, snapshots as tables plus watermark" +description = "Folding on a server (6j6v.vvw6): the reducers' described changes carried out against DynamoDB, the board's active, dated and typed indexes, next/blocked and a ranked list over them, snapshots as tables plus watermark" publish = false [features] @@ -37,3 +37,6 @@ aws-sdk-dynamodb = { version = "1", optional = true } tokio = { version = "1", features = ["rt-multi-thread", "macros", "time"] } # The records differential test (6j6v.t1ym) parses a plugin fixture that declares a custom field. toml = "0.8" +# The derived-keys differential test (6j6v.1c5s) registers its plugin fixture, so `plugin::load` +# validates it as it validates a consumer's plugin. +inventory = "0.3" diff --git a/crates/fold-ddb/src/board.rs b/crates/fold-ddb/src/board.rs index 2ffc850..8fbc37c 100644 --- a/crates/fold-ddb/src/board.rs +++ b/crates/fold-ddb/src/board.rs @@ -1,5 +1,5 @@ -//! The board's two indexes, kept beside the folded rows, and the reads that use them -//! (6j6v.vvw6 points 4 and 5). +//! The board's indexes, kept beside the folded rows, and the reads that use them +//! (6j6v.vvw6 points 4 and 5; `typed`, 6j6v.xkr7). //! //! # `active` //! @@ -27,6 +27,17 @@ //! one instant keep one order. A ticket without the instant sits under `"0"`, which sorts after //! every date — last, as the local lanes put it. [`dated`] reads one lane newest first. //! +//! # `typed` +//! +//! Sparse as well (6j6v.xkr7): a live item — not deleted, archived or not — with a type sits in +//! partition `#` ([`typed_partition`]) under its own sort key, so ONE Query lists the +//! live items of a type in id order: the `list` scope of a type the derived ranking keys rank by +//! (docs/specs/derived-ranking-keys.md §6). Like the `active` flag of a ticket it is a function of +//! the row alone, set whenever an op touches the row ([`refresh_item`]) — after the row, so a run +//! that dies in between sets it when it folds the op again. Generic: every typed item is listed, +//! whatever the plugin ranks, at one attribute per item row. A row's `typed` attribute never +//! names another row, so a retype or a delete moves nothing else. +//! //! # Who owns a label or a thread link //! //! `label_adds` and `thread_link_adds` key on the add's tag alone, so nothing in them is found by @@ -49,6 +60,19 @@ //! Only `contributes_to` adds get one: folding another edge's add costs nothing more, and folding //! an edge remove one `GetItem` of its add, to learn its kind. //! +//! # What an adjacency entry says +//! +//! Revision 3 (6j6v.1c5s) makes the `.adj##` entries self-describing, as the entries +//! under a ticket are: each copies the edge add's `from_id`, `to_id`, `kind`, `lamport` and `site` +//! and says whether the add is removed ([`REMOVED`]). Every edge add has one under each end, +//! whatever its kind and whatever the ends' activity, so ONE Query on a ticket's `.adj#` prefix +//! reads every edge touching it, in and out, with its presence — what the derived ranking keys read +//! ([`crate::records::with_ranking`]): inbound `contributes_to`, `mentions` both ways, the +//! `parent` edges a child's current parent is picked from. The entries keep their place and key; +//! only their attributes grow. Folding an edge add reads its remove (a `dep` or `parent` add read it +//! already, for its flag); folding an edge remove now reads the add and the remove and rewrites +//! both entries — two `GetItem`s and two writes, one when the edge is a loop. +//! //! # One writer per stream //! //! The rows a reducer describes converge whatever the order and however many folders write them. @@ -59,10 +83,11 @@ use crate::layout::{ adjacency_key, adjacency_prefix, contributes_adjacency_key, label_adjacency_key, - link_adjacency_key, row_key, table_prefix, ACTIVE, DATED, DATED_AT, MAX_KEY_BYTES, SK, + link_adjacency_key, row_key, table_prefix, typed_partition, ACTIVE, DATED, DATED_AT, + MAX_KEY_BYTES, SK, TYPED, }; -use crate::table::{text, Dated, Row, Table}; -use crate::write::{Cond, Write}; +use crate::table::{text, Dated, Outcome, Row, Table}; +use crate::write::{attribute_bytes, Cond, Write}; use nexus_flow_core::graph::{Board, Edge, Ticket}; use nexus_flow_core::model::EdgeKind; use nxs_foundation::change::{Cell, Change, Effect}; @@ -220,6 +245,29 @@ pub fn is_active(row: &Row) -> bool { ) } +/// The type a stored `items` row is listed under on the `typed` index (6j6v.xkr7): its type, when +/// it has one and is live — not deleted; archived or not. A type too long for an index partition +/// ([`MAX_PARTITION_BYTES`](crate::layout::MAX_PARTITION_BYTES)) is never folded: [`typed_partitions`] refuses its op first. +pub fn typed(row: &Row) -> Option<&str> { + if text(row, "deleted") == Some("1") { + return None; + } + text(row, "type") +} + +/// The `typed` partitions the item rows these changes write would be listed under — what +/// [`Folder`](crate::fold::Folder) checks against +/// [`MAX_PARTITION_BYTES`](crate::layout::MAX_PARTITION_BYTES) before an op writes +/// anything. +pub(crate) fn typed_partitions(stream: &str, changes: &[Change]) -> Vec { + changes + .iter() + .filter(|c| c.table == "items") + .filter_map(|c| effect_text(c, "type")) + .map(|ty| typed_partition(stream, ty)) + .collect() +} + /// The `dated` lane of a stored `items` row, and the instant it sorts by. pub fn lane(row: &Row) -> Option<(Dated, String)> { if text(row, "deleted") == Some("1") { @@ -246,8 +294,47 @@ pub fn dated_key(at: &str, id: &str) -> String { } } -/// Keep the indexes after one op's changes were carried out. -pub(crate) async fn after(table: &T, changes: &[Change]) -> Result<(), T::Error> { +/// The index attributes a stored `items` row of ticket `id` carries by the rules above — what +/// [`refresh_item`] leaves on it: `active`, `typed` and the `dated` pair, each only where it +/// applies. +pub(crate) fn index_attributes(stream: &str, id: &str, row: &Row) -> Vec<(String, Cell)> { + let mut cells = Vec::new(); + if is_active(row) { + cells.push((ACTIVE.to_string(), Cell::Text(stream.to_string()))); + } + if let Some(ty) = typed(row) { + cells.push((TYPED.to_string(), Cell::Text(typed_partition(stream, ty)))); + } + if let Some((l, at)) = lane(row) { + cells.push((DATED.to_string(), Cell::Text(l.partition(stream)))); + cells.push((DATED_AT.to_string(), Cell::Text(dated_key(&at, id)))); + } + cells +} + +/// What an `items` row weighs toward `MAX_ITEM_BYTES` once it carries the index attributes its +/// cells call for ([`index_attributes`]) instead of the ones it has — what +/// [`Folder::fold`](crate::fold::Folder::fold) checks before an op writes anything, so a row with +/// no room left for them is refused rather than left off an index (review of PR #55, Integrity +/// #4). +pub(crate) fn indexed_bytes(stream: &str, id: &str, row: &Row) -> usize { + let own: usize = row + .iter() + .filter(|(n, _)| ![ACTIVE, TYPED, DATED, DATED_AT].contains(&n.as_str())) + .map(|(n, c)| attribute_bytes(n, c)) + .sum(); + own + index_attributes(stream, id, row) + .iter() + .map(|(n, c)| attribute_bytes(n, c)) + .sum::() +} + +/// Keep the indexes after one op's changes were carried out. Returns the items whose row could not +/// take its index attributes ([`Outcome::TooLarge`]) — none, when the fold's size check ran first. +pub(crate) async fn after( + table: &T, + changes: &[Change], +) -> Result, T::Error> { let mut items = BTreeSet::new(); let mut added = BTreeSet::new(); let mut removed = BTreeSet::new(); @@ -272,19 +359,22 @@ pub(crate) async fn after(table: &T, changes: &[Change]) -> Result<(), _ => {} } } + let mut unindexed = Vec::new(); for id in items { - refresh_item(table, id).await?; + if refresh_item(table, id).await? == Some(Outcome::TooLarge) { + unindexed.push(id.to_string()); + } } for tag in added { link_edge(table, tag).await?; } for tag in removed { - refresh_edge(table, tag).await?; + link_edge(table, tag).await?; } for (at, tag) in owned_tags { refresh_owned(table, &OWNED[at], tag).await?; } - Ok(()) + Ok(unindexed) } /// Set ticket `id`'s index attributes from its row; when its activity flipped, refresh every edge @@ -293,10 +383,17 @@ pub(crate) async fn after(table: &T, changes: &[Change]) -> Result<(), /// The edges go FIRST, told the ticket's new activity, and the ticket's own flag after: the flag is /// what says whether the activity flipped, so a run that dies in between still sees the flip when /// it folds the op again, and redoes the fan-out (review of PR #24, Integrity #2). -pub(crate) async fn refresh_item(table: &T, id: &str) -> Result<(), T::Error> { +/// +/// Returns the outcome of the write of its index attributes, `None` when nothing changed — a +/// [`Outcome::TooLarge`] means the row is left with the attributes it had, which the caller must +/// report rather than drop (review of PR #55, Integrity #4). +pub(crate) async fn refresh_item( + table: &T, + id: &str, +) -> Result, T::Error> { let sk = item_key(id); let Some(row) = table.get(&sk).await? else { - return Ok(()); + return Ok(None); }; let active = is_active(&row); let was_active = row.contains_key(ACTIVE); @@ -304,7 +401,7 @@ pub(crate) async fn refresh_item(table: &T, id: &str) -> Result<(), T: for entry in table.query_prefix(&adjacency_prefix(id)).await? { if let Some(tag) = text(&entry, "tag") { if let Some(edge) = table.get(&tag_key("edge_adds", tag)).await? { - flag_edge(table, tag, edge, Some((id, active))).await?; + flag_edge(table, tag, edge, None, Some((id, active))).await?; } } } @@ -318,6 +415,14 @@ pub(crate) async fn refresh_item(table: &T, id: &str) -> Result<(), T: remove.push(ACTIVE.to_string()); } } + let typed_now = typed(&row).map(|ty| typed_partition(table.stream(), ty)); + let typed_was = text(&row, TYPED).map(str::to_string); + if typed_now != typed_was { + match typed_now { + Some(p) => set.push((TYPED.to_string(), Cell::Text(p))), + None => remove.push(TYPED.to_string()), + } + } let dated_now = lane(&row).map(|(l, at)| (l.partition(table.stream()), dated_key(&at, id))); let dated_was = match (row.get(DATED), row.get(DATED_AT)) { (Some(Cell::Text(p)), Some(Cell::Text(at))) => Some((p.clone(), at.clone())), @@ -336,9 +441,9 @@ pub(crate) async fn refresh_item(table: &T, id: &str) -> Result<(), T: } } if set.is_empty() && remove.is_empty() { - return Ok(()); + return Ok(None); } - table + let outcome = table .write(&Write { sk, set, @@ -346,49 +451,71 @@ pub(crate) async fn refresh_item(table: &T, id: &str) -> Result<(), T: condition: Some(Cond::RowPresent), }) .await?; - Ok(()) + Ok(Some(outcome)) } -/// Record that edge `tag` touches its ends, then set its flag. -pub(crate) async fn link_edge(table: &T, tag: &str) -> Result<(), T::Error> { - let Some(edge) = table.get(&tag_key("edge_adds", tag)).await? else { - return Ok(()); - }; - let ends: BTreeSet<&str> = [text(&edge, "from_id"), text(&edge, "to_id")] +/// The cells of edge add `edge` an adjacency entry copies beside its tag and [`REMOVED`] (index +/// revision 3, 6j6v.1c5s): its ends, its kind and its coordinate. +const EDGE_COPIES: [&str; 5] = ["from_id", "to_id", "kind", "lamport", "site"]; + +/// The adjacency entries of edge add `tag` — one under each end, the same entry twice for an edge +/// from a ticket to itself — from the add's row and whether it is removed. +async fn write_adjacency( + table: &T, + tag: &str, + edge: &Row, + removed: bool, +) -> Result<(), T::Error> { + let mut set = vec![("tag".to_string(), Cell::Text(tag.to_string()))]; + for column in EDGE_COPIES { + match edge.get(column) { + None | Some(Cell::Null) => {} + Some(cell) => set.push((column.to_string(), cell.clone())), + } + } + set.push((REMOVED.to_string(), Cell::Int(i64::from(removed)))); + let ends: BTreeSet<&str> = [text(edge, "from_id"), text(edge, "to_id")] .into_iter() .flatten() .collect(); for end in ends { table - .write(&Write::put( - adjacency_key(end, tag), - vec![("tag".to_string(), Cell::Text(tag.to_string()))], - )) + .write(&Write::put(adjacency_key(end, tag), set.clone())) .await?; } - flag_edge(table, tag, edge, None).await + Ok(()) } -/// Set edge `tag`'s flag from its row, its remove and its ends. -pub(crate) async fn refresh_edge(table: &T, tag: &str) -> Result<(), T::Error> { - match table.get(&tag_key("edge_adds", tag)).await? { - Some(edge) => flag_edge(table, tag, edge, None).await, - None => Ok(()), - } +/// Record that edge `tag` touches its ends — the entry under each end says what the edge is and +/// whether it is removed — then set its flag. Run when the add is folded and again when its remove +/// is; nothing while the add has not been folded, since the add writes both. +pub(crate) async fn link_edge(table: &T, tag: &str) -> Result<(), T::Error> { + let Some(edge) = table.get(&tag_key("edge_adds", tag)).await? else { + return Ok(()); + }; + let removed = table.get(&tag_key("edge_removes", tag)).await?.is_some(); + write_adjacency(table, tag, &edge, removed).await?; + flag_edge(table, tag, edge, Some(removed), None).await } -/// `known` is a ticket whose activity the caller knows ahead of its stored flag. +/// `removed` is whether the edge's remove is folded, when the caller has read it already; `known` +/// a ticket whose activity the caller knows ahead of its stored flag. async fn flag_edge( table: &T, tag: &str, edge: Row, + removed: Option, known: Option<(&str, bool)>, ) -> Result<(), T::Error> { let indexed = matches!( text(&edge, "kind").and_then(EdgeKind::parse), Some(EdgeKind::Dep | EdgeKind::Parent) ); - let mut flag = indexed && table.get(&tag_key("edge_removes", tag)).await?.is_none(); + let mut flag = indexed + && match removed { + Some(removed) => !removed, + None => table.get(&tag_key("edge_removes", tag)).await?.is_none(), + }; if flag { let mut any_active = false; for end in [text(&edge, "from_id"), text(&edge, "to_id")] @@ -432,11 +559,15 @@ async fn flag_edge( } /// Recompute every index attribute and adjacency entry from the rows — after a snapshot was -/// loaded, or to repair a stream two folders wrote at once. -pub async fn reindex(table: &T) -> Result<(), T::Error> { +/// loaded, or to repair a stream two folders wrote at once. Returns the items whose row could not +/// take its index attributes (past `MAX_ITEM_BYTES` with them), which keep the ones they had. +pub async fn reindex(table: &T) -> Result, T::Error> { + let mut unindexed = Vec::new(); for row in table.query_prefix(&table_prefix("items")).await? { if let Some(id) = text(&row, "id") { - refresh_item(table, id).await?; + if refresh_item(table, id).await? == Some(Outcome::TooLarge) { + unindexed.push(id.to_string()); + } } } for row in table.query_prefix(&table_prefix("edge_adds")).await? { @@ -451,7 +582,7 @@ pub async fn reindex(table: &T) -> Result<(), T::Error> { } } } - Ok(()) + Ok(unindexed) } /// The active tickets — their whole rows — and the board the library derives the lanes from. diff --git a/crates/fold-ddb/src/ddb.rs b/crates/fold-ddb/src/ddb.rs index 6089713..ffd46f1 100644 --- a/crates/fold-ddb/src/ddb.rs +++ b/crates/fold-ddb/src/ddb.rs @@ -1,25 +1,34 @@ //! [`DynamoDbTable`]: one stream's partition of the views table in DynamoDB, and [`create_table`], //! the table the contract in the crate doc describes. //! -//! Every request names the stream's partition key; the board's two reads name an index. There is +//! Every request names the stream's partition key — a `BatchGetItem` names it in every key; the +//! board's three index reads name an index. There is //! no Scan anywhere in this file, and the `dynamodb-local` CI job records every request a fold //! makes to prove it stays that way. //! //! The adapter is async and runs on the caller's runtime — a fold run in a Lambda awaits it //! directly, unlike the relay's op store, which bridges the SDK into its synchronous traits. -use crate::layout::{ACTIVE, ACTIVE_INDEX, DATED, DATED_AT, DATED_INDEX, PK, SK}; -use crate::table::{Dated, Outcome, Row, Table}; +use crate::layout::{ + typed_partition, ACTIVE, ACTIVE_INDEX, DATED, DATED_AT, DATED_INDEX, PK, SK, TYPED, TYPED_INDEX, +}; +use crate::table::{Dated, Outcome, Row, Table, BATCH_GET_KEYS}; use crate::write::Write; +use aws_sdk_dynamodb::config::AsyncSleep; use aws_sdk_dynamodb::error::{DisplayErrorContext, ProvideErrorMetadata}; use aws_sdk_dynamodb::operation::create_table::builders::CreateTableFluentBuilder; use aws_sdk_dynamodb::types::{ AttributeDefinition, AttributeValue, BillingMode, GlobalSecondaryIndex, KeySchemaElement, - KeyType, Projection, ProjectionType, ScalarAttributeType, + KeyType, KeysAndAttributes, Projection, ProjectionType, ScalarAttributeType, }; use aws_sdk_dynamodb::Client; use nxs_foundation::change::Cell; -use std::collections::HashMap; +use std::collections::{BTreeMap, BTreeSet, HashMap}; + +/// How often [`Table::get_many`] sends the keys a `BatchGetItem` left unprocessed again before it +/// gives up, waiting twice as long each time from [`UNPROCESSED_BACKOFF_MS`]. +const UNPROCESSED_RETRIES: u32 = 8; +const UNPROCESSED_BACKOFF_MS: u64 = 25; /// A request DynamoDB refused, or an item this crate cannot read. #[derive(Debug)] @@ -37,8 +46,8 @@ fn err(e: E) -> DdbError { DdbError(DisplayErrorContext(e).to_string()) } -/// The views table of `name`, as the deploying stack must provision it: keys, both sparse -/// indexes projecting every attribute, on-demand billing. Production code never sends it; the +/// The views table of `name`, as the deploying stack must provision it: keys, the three sparse +/// indexes (`active`, `dated`, `typed`) projecting every attribute, on-demand billing. Production code never sends it; the /// integration tests do, and a stack is checked against it. pub fn create_table(client: &Client, name: &str) -> CreateTableFluentBuilder { let s = |n: &str| { @@ -77,10 +86,12 @@ pub fn create_table(client: &Client, name: &str) -> CreateTableFluentBuilder { .attribute_definitions(s(ACTIVE)) .attribute_definitions(s(DATED)) .attribute_definitions(s(DATED_AT)) + .attribute_definitions(s(TYPED)) .key_schema(key(PK, KeyType::Hash)) .key_schema(key(SK, KeyType::Range)) .global_secondary_indexes(index(ACTIVE_INDEX, ACTIVE, SK)) .global_secondary_indexes(index(DATED_INDEX, DATED, DATED_AT)) + .global_secondary_indexes(index(TYPED_INDEX, TYPED, SK)) } /// One stream's partition of the views table `table`. @@ -206,6 +217,66 @@ impl Table for DynamoDbTable { out.item().map(from_item).transpose() } + async fn get_many(&self, sks: &[String]) -> Result, DdbError> { + let keys: Vec<&String> = sks.iter().collect::>().into_iter().collect(); + let mut rows = BTreeMap::new(); + for chunk in keys.chunks(BATCH_GET_KEYS) { + let mut pending: Vec> = + chunk.iter().map(|sk| self.key(sk)).collect(); + let mut retries = 0; + while !pending.is_empty() { + let request = KeysAndAttributes::builder() + .set_keys(Some(pending)) + .consistent_read(true) + .build() + .map_err(err)?; + let out = self + .client + .batch_get_item() + .request_items(&self.table, request) + .send() + .await + .map_err(err)?; + for item in out + .responses() + .and_then(|r| r.get(&self.table)) + .into_iter() + .flatten() + { + let row = from_item(item)?; + if let Some(Cell::Text(sk)) = row.get(SK) { + rows.insert(sk.clone(), row.clone()); + } + } + pending = out + .unprocessed_keys() + .and_then(|u| u.get(&self.table)) + .map(|k| k.keys().to_vec()) + .unwrap_or_default(); + if !pending.is_empty() { + // Throttled, or past the 16 MB a response carries: the rest again, after a + // pause that doubles, as DynamoDB asks of a caller. + if retries == UNPROCESSED_RETRIES { + return Err(DdbError(format!( + "BatchGetItem left {} keys unprocessed after {UNPROCESSED_RETRIES} \ + retries", + pending.len() + ))); + } + if let Some(sleep) = self.client.config().sleep_impl() { + sleep + .sleep(std::time::Duration::from_millis( + UNPROCESSED_BACKOFF_MS << retries, + )) + .await; + } + retries += 1; + } + } + } + Ok(rows) + } + async fn write(&self, write: &Write) -> Result { let e = write.expression(); let mut req = self @@ -303,4 +374,19 @@ impl Table for DynamoDbTable { ) .await } + + async fn query_typed(&self, item_type: &str) -> Result, DdbError> { + self.query_all( + Some(TYPED_INDEX), + "#t = :t", + &[("#t", TYPED)], + &[( + ":t", + AttributeValue::S(typed_partition(&self.stream, item_type)), + )], + false, + None, + ) + .await + } } diff --git a/crates/fold-ddb/src/fold.rs b/crates/fold-ddb/src/fold.rs index 6d0c283..5d8d5ad 100644 --- a/crates/fold-ddb/src/fold.rs +++ b/crates/fold-ddb/src/fold.rs @@ -13,7 +13,15 @@ //! //! When the recorded revisions differ from the reducers' — a release changed what the same ops fold //! to — [`Folder::is_current`] says so, and the run starts over: [`Folder::clear`] the stream's -//! rows, then fold from relay position 0. +//! rows, then fold from relay position 0. The clear leaves a sentinel watermark naming the position +//! the refold has to reach, and readers answer `NotCurrent` / `Refolding` until it has. +//! +//! **No mixed fold versions.** `is_current` is equality, not order: a build with OLDER rules that +//! meets a stream folded by newer ones also finds it not current, and clears and refolds it back. +//! Two builds with different revisions folding the same streams — a rolling deploy, a rollback with +//! a newer build still running — would refold every stream back and forth. Roll a revision bump +//! out to every fold run at once, and roll back the same way (a fold run that builds anew beside +//! the served build, as one that keeps a namespace per build, ping-pongs builds instead). //! //! The rules this crate adds beside the reducers' rows — the board's index flags and the entries it //! keeps under a ticket — have a revision of their own, [`INDEX_REVISION`], recorded in the same @@ -21,7 +29,11 @@ //! bumps it, and every stream folded before is no longer current and is folded again the same way. //! Revision 1 (6j6v.t1ym) adds the `.ladj#`/`.tadj#` entries; a stream folded before it has none, //! so its watermark names no index revision at all and the next run refolds it. Revision 2 -//! (6j6v.15ed) adds the `.cadj#` entries, a ticket's outgoing `contributes_to` edges. +//! (6j6v.15ed) adds the `.cadj#` entries, a ticket's outgoing `contributes_to` edges. Revision 3 +//! (6j6v.1c5s) makes the `.adj#` entries self-describing — ends, kind, coordinate, presence — for +//! the derived ranking keys ([`crate::records::with_ranking`]), and sets the `typed` index +//! attribute on every live item row (6j6v.xkr7, [`crate::records::list_board`]). Both ship in one +//! release: no stream was ever folded by revision 3 without the attribute. //! //! # What the server cannot see //! @@ -32,8 +44,11 @@ //! coordinate. use crate::board; -use crate::layout::{Layout, LayoutError, MAX_ITEM_BYTES, MAX_KEY_BYTES, WATERMARK_KEY}; -use crate::table::{Outcome, Table}; +use crate::layout::{ + table_prefix, Layout, LayoutError, MAX_ITEM_BYTES, MAX_KEY_BYTES, MAX_PARTITION_BYTES, PK, SK, + WATERMARK_KEY, +}; +use crate::table::{text, Outcome, Table}; use crate::write::{plan, Cond, Write}; use nxs_foundation::change::{Cell, Change, Wins}; use nxs_foundation::model::Op; @@ -44,9 +59,14 @@ use std::collections::BTreeMap; /// (see the module doc). No reducer has this domain. pub const INDEX_DOMAIN: &str = "index"; +/// The watermark attribute [`Folder::clear`] sets: [`Watermark::refold_until`]. +pub const REFOLD_UNTIL: &str = "refold_until"; + /// The revision of those rules. 1: the entries under a ticket for its label and thread link adds /// (6j6v.t1ym). 2: the entries under a ticket for its outgoing `contributes_to` edges (6j6v.15ed). -pub const INDEX_REVISION: i64 = 2; +/// 3: the adjacency entries copy their edge's ends, kind and coordinate and say whether it is +/// removed (6j6v.1c5s), and every live item row carries its `typed` index partition (6j6v.xkr7). +pub const INDEX_REVISION: i64 = 3; /// The platform's reducers — the board's, the chat's and the memory's. pub fn platform_reducers() -> Vec> { @@ -72,6 +92,10 @@ pub struct Watermark { pub folded_through: i64, /// Each reducer's fold revision, by domain. pub revisions: BTreeMap, + /// Set by [`Folder::clear`]: the relay position a refold has to reach before the stream is + /// served again — readers answer `RecordsError::Refolding` while `folded_through` is below it + /// (review of PR #55, Integrity #2). `None` on a stream never cleared. + pub refold_until: Option, } /// What [`Folder::fold`] did with an op. @@ -158,11 +182,15 @@ impl Folder { /// Fold one op: carry out its changes, then keep the board's indexes. /// - /// An op the table cannot hold — a key past `MAX_KEY_BYTES`, a row past `MAX_ITEM_BYTES` — is + /// An op the table cannot hold — a key past `MAX_KEY_BYTES`, a type whose `typed` index + /// partition is past `MAX_PARTITION_BYTES`, a row past `MAX_ITEM_BYTES`, an item row that would + /// pass it once it carries its index attributes (`active`, `typed`, `dated`; one `GetItem` per + /// item row the op writes, to weigh it) — is /// [`Folded::Refused`] before it writes anything, rather than failing the batch for ever: the - /// relay keeps it, as it keeps any op a server does not fold. A row that only outgrows the limit - /// through earlier ops refuses the one write that would break it ([`Outcome::TooLarge`]); the - /// op's other writes stand. That last case is the one place the result depends on the order ops + /// relay keeps it, as it keeps any op a server does not fold. An item row that only outgrows + /// the limit through earlier ops is caught by that weighing too, so the whole op is refused; + /// any other row refuses the one write that would break it ([`Outcome::TooLarge`]), and the + /// op's other writes stand. Those cases are the one place the result depends on the order ops /// arrive in — which op's cell made the row too large — and a local replica, which has no item /// limit, holds all of it. pub async fn fold(&self, table: &T, op: &Op) -> Result> { @@ -193,21 +221,50 @@ impl Folder { ))); } } + if let Some(p) = board::typed_partitions(table.stream(), &changes) + .into_iter() + .find(|p| p.len() > MAX_PARTITION_BYTES) + { + return Ok(refuse(format!( + "a type whose `typed` index partition has {} bytes, past the {MAX_PARTITION_BYTES} \ + a partition key may have", + p.len() + ))); + } if let Some(w) = writes.iter().find(|w| w.bytes() > MAX_ITEM_BYTES) { return Ok(refuse(format!( "a row of {} bytes, past the {MAX_ITEM_BYTES} an item may have", w.bytes() ))); } + // An item row takes its index attributes in a write of their own, after the row + // (`board::refresh_item`): one the row has no room for would leave the item off an index + // in silence. So the item rows this op writes are weighed first — stored row, this op's + // writes, and the index attributes it would then carry — and the op is refused before it + // writes anything (review of PR #55, Integrity #4). + for (id, bytes) in self.item_bytes(table, &writes).await? { + if bytes > MAX_ITEM_BYTES { + return Ok(refuse(format!( + "item {id} would weigh {bytes} bytes with its index attributes, past the \ + {MAX_ITEM_BYTES} an item may have" + ))); + } + } let mut too_large = Vec::new(); for write in &writes { if table.write(write).await.map_err(FoldError::Table)? == Outcome::TooLarge { too_large.push(write.sk.clone()); } } - board::after(table, &changes) - .await - .map_err(FoldError::Table)?; + // Past the check above only where DynamoDB counts a few bytes more than `attribute_bytes`: + // reported, never dropped. + too_large.extend( + board::after(table, &changes) + .await + .map_err(FoldError::Table)? + .into_iter() + .map(|id| format!("the index attributes of item {id}")), + ); if too_large.is_empty() { Ok(Folded::Folded) } else { @@ -218,6 +275,50 @@ impl Folder { } } + /// What each item row `writes` touch would weigh after them, with the index attributes it + /// would then carry ([`board::indexed_bytes`]): one `GetItem` per such row. Every write is + /// counted as applied, its condition aside — so a write that would be kept can only make the + /// estimate larger, never let a row past the limit. + async fn item_bytes( + &self, + table: &T, + writes: &[Write], + ) -> Result, FoldError> { + let prefix = table_prefix("items"); + let mut rows: BTreeMap<&str, crate::table::Row> = BTreeMap::new(); + for write in writes.iter().filter(|w| w.sk.starts_with(&prefix)) { + if !rows.contains_key(write.sk.as_str()) { + let stored = table + .get(&write.sk) + .await + .map_err(FoldError::Table)? + .unwrap_or_else(|| { + crate::table::Row::from([ + (PK.to_string(), Cell::Text(table.stream().to_string())), + (SK.to_string(), Cell::Text(write.sk.clone())), + ]) + }); + rows.insert(write.sk.as_str(), stored); + } + if let Some(row) = rows.get_mut(write.sk.as_str()) { + for (a, c) in &write.set { + row.insert(a.clone(), c.clone()); + } + for a in &write.remove { + row.remove(a); + } + } + } + Ok(rows + .values() + .map(|row| { + let id = text(row, "id").unwrap_or_default().to_string(); + let bytes = board::indexed_bytes(table.stream(), &id, row); + (id, bytes) + }) + .collect()) + } + /// Fold a batch of ops pulled from the relay — each with its relay position — and move the /// watermark to the highest position among them. Returns the ops it refused, for the caller to /// log; they do not hold the watermark back. @@ -268,6 +369,36 @@ impl Folder { .map_err(FoldError::Table) } + /// A watermark that names no revision — so no reader serves the stream ([`Folder::clear`], + /// `snapshot::import`) — at `folded_through`, with the position a refold must reach. + pub(crate) fn sentinel(folded_through: i64, until: Option) -> Write { + let mut set = vec![("folded_through".to_string(), Cell::Int(folded_through))]; + set.extend(until.map(|u| (REFOLD_UNTIL.to_string(), Cell::Int(u)))); + Write { + sk: WATERMARK_KEY.to_string(), + set, + remove: Vec::new(), + condition: None, + } + } + + /// This folder's watermark at `through`, written whatever the stored one says, and without a + /// refold position — what `snapshot::import` ends with. + pub(crate) fn stamp(&self, through: i64) -> Write { + let mut set = vec![("folded_through".to_string(), Cell::Int(through))]; + set.extend( + self.revisions() + .into_iter() + .map(|(domain, rev)| (format!("revision_{domain}"), Cell::Int(rev))), + ); + Write { + sk: WATERMARK_KEY.to_string(), + set, + remove: vec![REFOLD_UNTIL.to_string()], + condition: None, + } + } + /// The stream's watermark, `None` before its first batch. pub async fn watermark(table: &T) -> Result, T::Error> { Ok(table.get(WATERMARK_KEY).await?.map(|row| { @@ -275,6 +406,7 @@ impl Folder { for (attr, cell) in row { match (attr.as_str(), cell) { ("folded_through", Cell::Int(v)) => mark.folded_through = v, + (REFOLD_UNTIL, Cell::Int(v)) => mark.refold_until = Some(v), (a, Cell::Int(v)) if a.starts_with("revision_") => { mark.revisions.insert(a["revision_".len()..].to_string(), v); } @@ -291,18 +423,42 @@ impl Folder { mark.revisions == self.revisions() } - /// Delete every row of the stream — the views, the adjacency entries and the watermark, the - /// watermark LAST: a clear that dies half way leaves a stream whose watermark still names the old - /// revisions, so the next run clears it again instead of taking the leftovers for a fresh stream - /// (review of PR #24, Integrity #3). - pub async fn clear(table: &T) -> Result<(), T::Error> { - for row in table.query_prefix("").await? { - if let Some(Cell::Text(sk)) = row.get(crate::layout::SK) { + /// Delete every row of the stream — the views and the adjacency entries — for a refold from + /// position 0 that has to reach relay position `until` (the stream's head when the refold + /// begins) before the stream is served again. + /// + /// The watermark is not deleted: it is rewritten FIRST, as a sentinel that names no revision, + /// `folded_through` 0 and `refold_until` = `until`. So readers answer `NotCurrent` from the + /// moment the clear begins (review of PR #55, Integrity #2) — a cleared, half-cleared or + /// half-refolded stream is never served as if whole — and a clear that dies half way leaves a + /// watermark that is not current, so the next run clears it again instead of taking the + /// leftovers for a fresh stream (review of PR #24, Integrity #3). The refold's first batch + /// writes this build's revisions ([`Folder::advance`]), and from then on the readers answer + /// `Refolding` until `folded_through` reaches `until`; nothing has to be written for that, so a + /// refold that dies and is resumed, or folds a batch twice, ends served all the same. `until` + /// 0 serves the stream from its first refolded batch on, as before revisions were gated. + pub async fn clear(table: &T, until: i64) -> Result<(), T::Error> { + let rows = table.query_prefix("").await?; + let stale: Vec = rows + .iter() + .find(|r| text(r, crate::layout::SK) == Some(WATERMARK_KEY)) + .map(|mark| { + mark.keys() + .filter(|a| a.starts_with("revision_")) + .cloned() + .collect() + }) + .unwrap_or_default(); + let mut sentinel = Folder::sentinel(0, Some(until)); + sentinel.remove = stale; + table.write(&sentinel).await?; + for row in rows { + if let Some(sk) = text(&row, crate::layout::SK) { if sk != WATERMARK_KEY { table.delete(sk).await?; } } } - table.delete(WATERMARK_KEY).await + Ok(()) } } diff --git a/crates/fold-ddb/src/layout.rs b/crates/fold-ddb/src/layout.rs index ffd4943..73475e6 100644 --- a/crates/fold-ddb/src/layout.rs +++ b/crates/fold-ddb/src/layout.rs @@ -26,18 +26,26 @@ pub const ACTIVE: &str = "nxf_active"; pub const DATED: &str = "nxf_dated"; /// The sort key of the `dated` index: the instant the ticket entered its lane. pub const DATED_AT: &str = "nxf_dated_at"; +/// The partition key of the `typed` index (6j6v.xkr7): `#` ([`typed_partition`]), +/// present only on a live item row — not deleted, archived or not — that has a type. +pub const TYPED: &str = "nxf_typed"; /// The `active` index. pub const ACTIVE_INDEX: &str = "active"; /// The `dated` index. pub const DATED_INDEX: &str = "dated"; +/// The `typed` index (6j6v.xkr7). +pub const TYPED_INDEX: &str = "typed"; /// The longest sort key — and `dated` index sort key — DynamoDB stores, in bytes. pub const MAX_KEY_BYTES: usize = 1024; +/// The longest partition key value DynamoDB stores — on the table or on an index — in bytes. Only +/// the `typed` index's can come near it: it carries an item's type ([`typed_partition`]). +pub const MAX_PARTITION_BYTES: usize = 2048; /// The largest item DynamoDB stores, in bytes, counting attribute names and values. pub const MAX_ITEM_BYTES: usize = 400 * 1024; /// Attribute names no view column may take: the table's own keys and the index attributes. -pub const RESERVED: [&str; 5] = [PK, SK, ACTIVE, DATED, DATED_AT]; +pub const RESERVED: [&str; 6] = [PK, SK, ACTIVE, DATED, DATED_AT, TYPED]; /// The fold's own bookkeeping, outside every view table: a view table's name is an identifier and /// cannot start with `.`. @@ -69,6 +77,13 @@ pub fn row_key(table: &str, key: &[(String, Cell)]) -> Result#`, the +/// type with `%` and `#` escaped as a key value is, so two streams' types can never share a +/// partition (`a` + `b#c` and `a#b` + `c`). +pub fn typed_partition(stream: &str, item_type: &str) -> String { + format!("{stream}#{}", escape(item_type)) +} + /// The prefix every row of `table` starts with. pub fn table_prefix(table: &str) -> String { format!("{table}#") diff --git a/crates/fold-ddb/src/lib.rs b/crates/fold-ddb/src/lib.rs index 868b37b..59e2c23 100644 --- a/crates/fold-ddb/src/lib.rs +++ b/crates/fold-ddb/src/lib.rs @@ -1,7 +1,8 @@ //! Folding on a server (6j6v.vvw6): the reducers' [described changes](nxs_foundation::change) //! carried out as conditional writes against one table per deployment, the board's `active` and //! `dated` indexes kept beside the rows, next/blocked derived by the same library a local replica -//! uses, and snapshots that are tables plus watermark, without the log. +//! uses, and snapshots that are tables plus watermark, without the log; a type's live items listed +//! by a third index, `typed`, for a ranked hosted `list`. //! //! The pieces, in the order a fold run uses them: [`fold::Folder`] maps each op through the //! platform's reducers and [`write::plan`] turns every change into a [`write::Write`]; @@ -22,23 +23,42 @@ //! | sort key | `sk` | S | `#`, `.adj#`/`.ladj#`/`.tadj##`, `.meta#watermark` | //! | GSI `active` | `nxf_active` / `sk` | S/S | sparse: active tickets and the present dep/parent edges touching one | //! | GSI `dated` | `nxf_dated` / `nxf_dated_at` | S/S | sparse: `#closed` and `#archived`, by `␟` | +//! | GSI `typed` | `nxf_typed` / `sk` | S/S | sparse: `#` on every live item row (not deleted; archived included) | //! -//! Both indexes project ALL attributes, so a lane is one Query and no follow-up reads. The lanes +//! The `.adj##` entries are plain rows in the stream's partition, on no index. Since +//! index revision 3 (6j6v.1c5s) each also carries its edge's `from_id`, `to_id`, `kind`, `lamport`, +//! `site` and a `removed` flag — attributes, not keys. +//! +//! Every index projects ALL attributes, so a lane is one Query and no follow-up reads. +//! +//! **The `typed` index (6j6v.xkr7)** is the one change to this contract since `active` and `dated` +//! (docs/specs/derived-ranking-keys.md §6, owner decision §10.4): one Query lists the live items of +//! a type, the `list` scope a hosted `list` and the derived ranking keys rank over +//! ([`records::list_board`]). Its partition key `nxf_typed` is `#`, the type escaped +//! as a key value is (`%` → `%25`, `#` → `%23`; [`layout::typed_partition`]); its sort key is the +//! table's `sk`. The fold sets it on every item row whose type is set and which is not deleted — +//! closed and archived ones included — and removes it on a delete, written after the row like the +//! other index attributes ([`board`], "`typed`"). The deploying stack adds the GSI FIRST: a fold +//! against a table without it still writes the attribute, which costs nothing and breaks nothing; +//! only the read — a `list` through [`records::list_board`], a `next` whose keys read a +//! `[ranking.list]` rank — needs the index. A type whose partition would pass DynamoDB's 2,048 +//! bytes for an index key is refused at fold time ([`layout::MAX_PARTITION_BYTES`]). The lanes //! need nothing else; the RECORDS a lane is shown with carry joins from other tables, which //! [`records`] reads with a fixed number of requests per active ticket. Billing mode //! is the deployment's call. No local secondary index: without one, DynamoDB sets no limit on the //! size of a stream's partition (the 10 GB item-collection limit applies only to tables with an LSI). //! What does bound a stream is the throughput of ONE partition key: a stream's `active` and `dated` -//! entries each share one index partition — a write-heavy stream is limited to that partition's -//! throughput (1,000 writes per second). +//! entries each share one index partition, its items of one type one `typed` partition — a +//! write-heavy stream is limited to that partition's throughput (1,000 writes per second). //! //! Every row of view table `t` carries its columns as attributes of the same name ([`layout`]); a //! NULL cell is DynamoDB's NULL, an integer a number, text a string. //! //! # Freshness //! -//! A row read (`GetItem`, a Query on the table) is strongly consistent. The two index reads — -//! `board::select` and `board::dated` — are eventually consistent, as every DynamoDB GSI read is: +//! A row read (`GetItem`, a Query on the table) is strongly consistent. The index reads — +//! `board::select`, `board::dated` and the `typed` Query of [`records::list_board`] — are eventually +//! consistent, as every DynamoDB GSI read is: //! right after a fold they may answer the state before it, for as long as the index takes to catch //! up (typically well under a second). The lanes are the local lanes of THAT state. A reader that //! must see its own write reads the ticket's row instead. @@ -56,12 +76,25 @@ //! # Cost //! //! A change is one `UpdateItem`. An item field op makes three changes (the cell and the ticket's -//! two instants) plus one `GetItem` for the index, and one more write when the ticket's lanes move. +//! two instants) plus two `GetItem`s — one before, to weigh the row with the index attributes it +//! would then carry against the 400 KB limit, one after, for the index — and one more write when +//! the ticket's lanes move. //! A flip of a ticket's activity adds one Query and, per edge touching it, at most three reads and //! one write. A label or thread link add, or its remove, adds two `GetItem`s and at most one write //! for the entry under the ticket (`board`, "Who owns a label or a thread link"). Reading the //! active tickets' records costs `4n + 1` requests for `n` tickets, plus one per parent outside -//! the selection, at most [`records::READ_CONCURRENCY`] tickets at a time ([`records`]). No request +//! the selection, at most [`records::READ_CONCURRENCY`] tickets at a time ([`records`]). Folding an +//! edge remove reads the add and the remove and rewrites the entry under each end. The graph the +//! derived ranking keys read ([`records::with_ranking`]) costs one Query per active ticket and, per +//! inactive neighbour, up to three Queries, their rows read by `BatchGetItem` (100 keys a request). +//! A hosted `list` ([`records::list_board`]) costs one `typed` Query per listed type; one that is +//! not ranked keeps and cuts its rows first and reads three Queries per row of its page plus one +//! `BatchGetItem` for their timestamps, however many items the stream holds; a ranked one reads +//! three Queries per live item of its types — the scope its rank is computed over — and its point +//! reads by `BatchGetItem`. No read caps its own requests: a server bounds one request by the +//! table it hands in (E4 spec §5.1). The `typed` index costs a fold at most one write per item +//! op, and only when the op gains, changes or loses the item's partition — a create, a retype, a +//! delete — in the same write as the item's other index attributes when they move too. No request //! is a Scan: every read names its partition key, the board's through an index. The //! `dynamodb-local` CI job counts the requests of a fold and of a records read, and proves it. //! @@ -70,7 +103,10 @@ //! What this crate writes beside the reducers' rows — index flags, adjacency entries, the entries //! under a ticket — has its own revision in the watermark ([`fold::INDEX_REVISION`]). A stream //! folded by an older one is not current ([`fold::Folder::is_current`]) and is refolded the way a -//! stream of an older reducer revision is; [`records::active_board`] refuses it until then. +//! stream of an older reducer revision is. Every reader in [`records`] refuses it from the moment +//! the clear begins until the refold reaches the head the clear named ([`fold::Folder::clear`]): +//! `NotCurrent` while the watermark names no revision, `Refolding` after. Fold versions are not +//! mixed: a build with older rules meets a newer stream as not current, too (see [`fold`]). pub mod board; pub mod chunks; diff --git a/crates/fold-ddb/src/mem.rs b/crates/fold-ddb/src/mem.rs index 09178aa..5462097 100644 --- a/crates/fold-ddb/src/mem.rs +++ b/crates/fold-ddb/src/mem.rs @@ -7,8 +7,8 @@ //! It counts the requests it answers ([`Requests`]), so the cost of a fold — the flag fan-out above //! all — is measured on the same code path the DynamoDB table runs. -use crate::layout::{ACTIVE, DATED, DATED_AT, MAX_ITEM_BYTES, PK, SK}; -use crate::table::{Dated, Outcome, Row, Table}; +use crate::layout::{typed_partition, ACTIVE, DATED, DATED_AT, MAX_ITEM_BYTES, PK, SK, TYPED}; +use crate::table::{Dated, Outcome, Row, Table, BATCH_GET_KEYS}; use crate::write::{attribute_bytes, beats, Cond, Write}; use nxs_foundation::change::Cell; use std::collections::BTreeMap; @@ -19,6 +19,9 @@ use std::sync::Mutex; #[derive(Debug, Clone, Copy, Default, PartialEq, Eq)] pub struct Requests { pub gets: u64, + /// `BatchGetItem` requests: one per [`BATCH_GET_KEYS`] distinct keys of a + /// [`get_many`](Table::get_many). + pub batch_gets: u64, pub writes: u64, pub deletes: u64, pub queries: u64, @@ -94,6 +97,16 @@ impl Table for MemTable { Ok(self.rows.lock().unwrap().get(sk).cloned()) } + async fn get_many(&self, sks: &[String]) -> Result, Infallible> { + let keys: std::collections::BTreeSet<&String> = sks.iter().collect(); + self.count(|r| r.batch_gets += keys.len().div_ceil(BATCH_GET_KEYS) as u64); + let rows = self.rows.lock().unwrap(); + Ok(keys + .into_iter() + .filter_map(|sk| rows.get(sk).map(|r| (sk.clone(), r.clone()))) + .collect()) + } + async fn write(&self, write: &Write) -> Result { self.count(|r| r.writes += 1); let mut rows = self.rows.lock().unwrap(); @@ -176,6 +189,20 @@ impl Table for MemTable { rows.truncate(limit.unwrap_or(usize::MAX)); Ok(rows) } + + async fn query_typed(&self, item_type: &str) -> Result, Infallible> { + self.count(|r| r.index_queries += 1); + let partition = Cell::Text(typed_partition(&self.stream, item_type)); + // The rows are kept by sort key: the index's own order. + Ok(self + .rows + .lock() + .unwrap() + .values() + .filter(|r| r.get(TYPED) == Some(&partition)) + .cloned() + .collect()) + } } /// Run a future that never waits — every [`MemTable`] future — to completion on this thread. diff --git a/crates/fold-ddb/src/records.rs b/crates/fold-ddb/src/records.rs index cb930d8..bb0d35e 100644 --- a/crates/fold-ddb/src/records.rs +++ b/crates/fold-ddb/src/records.rs @@ -34,6 +34,21 @@ //! a time, so a board's latency is about `n / READ_CONCURRENCY` round trips, and no board sends more //! than that many requests at once. //! +//! One more is read only for a plugin that ranks `next` by derived ranking keys: +//! [`with_ranking`] fills the board's [`RankingGraph`] — the edges around the active tickets and +//! their live neighbours — from the adjacency entries (index revision 3, 6j6v.1c5s), and the live +//! items of a type whose `[ranking.list]` rank a key reads from the `typed` index (6j6v.xkr7). Its +//! cost is in its own doc. Where it reads many rows by key it sends `BatchGetItem`s +//! ([`Table::get_many`], 100 keys a request) instead of one `GetItem` each. +//! +//! # A hosted `list` +//! +//! [`list_board`] reads what the facade's store-free `list` ranks and projects (6j6v.xkr7): the +//! live, unarchived items of the listed types off the `typed` index, with the joins a `list` +//! record and its filters carry, `next`'s board when a key reads a rank `next` gives, and the +//! graph. The server asks [`read::list_reads`](nexus_flow_facade::read::list_reads) what to read +//! for its query; its cost is in its own doc. +//! //! # Two points in time //! //! The selection is an index read, eventually consistent; every join is a row read, strongly @@ -47,23 +62,36 @@ //! # A stream folded by older rules //! //! The entries under a ticket came with index revision 1, the `.cadj#` entries with revision 2 -//! (6j6v.15ed). A stream whose watermark names an older -//! one — or none, folded before revisions were recorded for the index — has no entries, and its -//! records would come back without labels or conversations. [`active_board`] refuses such a stream -//! ([`RecordsError::NotCurrent`]); the fold run refolds it ([`Folder::is_current`]), after which it -//! reads normally. +//! (6j6v.15ed), the self-describing `.adj#` entries and the `typed` index attribute with revision 3 +//! (6j6v.1c5s, 6j6v.xkr7). A stream whose watermark names an older one — or none, folded before +//! revisions were recorded for the index — lacks them, and its records would come back without +//! labels or conversations, its list empty. [`active_board`], [`with_ranking`] and [`list_board`] +//! refuse such a stream ([`RecordsError::NotCurrent`]); the fold run refolds it +//! ([`Folder::is_current`]), after which it reads normally. +//! +//! A refold is refused for its whole length, not only until it starts: [`Folder::clear`] leaves a +//! sentinel watermark that names no revision (`NotCurrent`), and once the refold has written this +//! build's revisions the readers answer [`RecordsError::Refolding`] until it reaches the relay +//! position the clear named. A stream with no watermark at all was never folded, and reads as the +//! empty board it is. //! //! [`Folder::is_current`]: crate::fold::Folder::is_current +//! [`Folder::clear`]: crate::fold::Folder::clear -use crate::board::{int, item_key, Selection, REMOVED}; +use crate::board::{int, is_active, item_key, select, Selection, REMOVED}; use crate::fold::{Folder, INDEX_DOMAIN, INDEX_REVISION}; use crate::layout::{ - contributes_adjacency_prefix, label_adjacency_prefix, link_adjacency_prefix, row_key, + adjacency_prefix, contributes_adjacency_prefix, label_adjacency_prefix, link_adjacency_prefix, + row_key, table_prefix, }; use crate::table::{text, Row, Table}; use futures_util::stream::{self, StreamExt, TryStreamExt}; -use nexus_flow_core::model::{ItemRow, LinkWeight}; -use nexus_flow_facade::read::{ActiveBoard, ActiveTicket, ParentRef}; +use nexus_flow_core::graph::Edge; +use nexus_flow_core::model::{EdgeKind, ItemRow, LinkWeight}; +use nexus_flow_facade::read::{ + self, ActiveBoard, ActiveTicket, ListBoard, ListItem, ListReads, ListSelect, Neighbour, + ParentRef, RankingGraph, RankingReads, +}; use nxs_foundation::change::Cell; use std::collections::{BTreeMap, BTreeSet}; @@ -80,6 +108,10 @@ pub enum RecordsError { /// watermark names, `None` when it names none). Its records would lack joins; the fold run /// refolds it. NotCurrent { found: Option }, + /// The stream is being folded again from position 0 ([`Folder::clear`]) and has not yet + /// reached `until`, the relay position the refold has to reach before it is served: its + /// tables hold part of its history only. + Refolding { folded_through: i64, until: i64 }, } impl std::fmt::Display for RecordsError { @@ -91,6 +123,14 @@ impl std::fmt::Display for RecordsError { "the stream was folded by index revision {found:?}, this build reads revision \ {INDEX_REVISION}; it has to be folded again" ), + RecordsError::Refolding { + folded_through, + until, + } => write!( + f, + "the stream is being folded again, through relay position {folded_through} of \ + {until}; it is served once the refold reaches {until}" + ), } } } @@ -110,12 +150,7 @@ pub async fn active_board( table: &T, selection: Selection, ) -> Result> { - if let Some(mark) = Folder::watermark(table).await? { - let found = mark.revisions.get(INDEX_DOMAIN).copied(); - if found != Some(INDEX_REVISION) { - return Err(RecordsError::NotCurrent { found }); - } - } + check_current(table).await?; let Selection { items, board, @@ -206,6 +241,579 @@ async fn contributes_to(table: &T, id: &str) -> Result, T: Ok(targets.into_iter().collect()) } +/// One present edge as its adjacency entry tells it (index revision 3): the add's ends, kind and +/// coordinate (`-1` where the add has none, as `present_parent` reads it). +#[derive(Debug, Clone)] +struct EdgeEntry { + from: String, + to: String, + kind: EdgeKind, + lamport: i64, + site: i64, +} + +/// The present edges touching ticket `id`, by tag: one Query on its `.adj#` prefix. Only entries +/// that name `id` as an end count. +async fn present_edges( + table: &T, + id: &str, +) -> Result, T::Error> { + Ok(present_entries(table, &adjacency_prefix(id)) + .await? + .iter() + .filter_map(|e| { + let entry = EdgeEntry { + from: text(e, "from_id")?.to_string(), + to: text(e, "to_id")?.to_string(), + kind: text(e, "kind").and_then(EdgeKind::parse)?, + lamport: int(e, "lamport").unwrap_or(-1), + site: int(e, "site").unwrap_or(-1), + }; + Some((text(e, "tag")?.to_string(), entry)) + }) + // An entry filed under `id` names it as an end. One that does not — an add whose tag was + // reused with other ends, leaving the entry under its old end behind — is not an edge of + // `id`, whatever its copy says (review of PR #55, Integrity #5). + .filter(|(_, e)| e.from == id || e.to == id) + .collect()) +} + +/// The current parent of `id` among the present edges touching it — `present_parent`'s pick: of its +/// present outgoing `parent` edges, the one with the highest `(lamport, site, to_id)`. +fn current_parent(id: &str, edges: &BTreeMap) -> Option { + edges + .values() + .filter(|e| e.kind == EdgeKind::Parent && e.from == id) + .max_by(|a, b| (a.lamport, a.site, &a.to).cmp(&(b.lamport, b.site, &b.to))) + .map(|e| e.to.clone()) +} + +/// `board` with its [`RankingGraph`] for `reads` (6j6v.1c5s): the graph the derived ranking keys of +/// `[ranking.next]` read, from the folded tables — what [`read::with_ranking`] reads from a store, +/// edge for edge and neighbour for neighbour. A server calls it when +/// [`read::next_ranking_reads`] names reads for its plugin, so a plugin without derived `next` keys +/// pays nothing for it. The stream must be current, as for [`active_board`] — the entries read here +/// came with index revision 3 — and it is asked again here ([`RecordsError::NotCurrent`], +/// [`RecordsError::Refolding`]): a revision-2 entry says nothing of its edge, and would read as no +/// edge at all. +/// +/// | read | per | +/// |-----------------------------------------------|-------------------------------------------| +/// | one `GetItem` of `.meta#watermark` | board | +/// | one Query on `.adj##` | active ticket | +/// | one Query on the `typed` index | type in `reads.list_types` (6j6v.xkr7) | +/// | one Query on `.adj##` | live, unarchived item of such a type that is not active | +/// | its row, by `BatchGetItem` | other end of such an edge that is not a subject — `⌈p / 100⌉` requests | +/// | one Query on `.adj##` (its parent) | such end that is live | +/// | one Query on `.ladj##`, `custom_fields##` | such end and listed item, when `reads` asks for labels, custom values | +/// +/// So `n` active tickets with `p` distinct inactive neighbours, `q` of them live, cost `n + q` +/// Queries, one `GetItem` and `⌈p / 100⌉` `BatchGetItem`s — `n + 3q` Queries with labels and +/// custom values — at most +/// [`READ_CONCURRENCY`] items at a time, and nothing grows with an item's history: an entry says +/// whether its add is removed. Every edge touching a subject is read once, from one end's entries, +/// whatever the plugin's keys — the edge pass the store path runs once per query. A `next` key +/// that reads the rank of a type only `[ranking.list]` orders adds, per such type, one index Query +/// and its live items as subjects ([`RankingReads::list_types`]) — one Query on `.adj#`, and one each +/// on `.ladj#` / `custom_fields#` when the keys read them, per live, unarchived item of the type, +/// however few the answer ranks: the rank is computed over the type's whole `list` scope +/// (docs/specs/derived-ranking-keys.md §12, decision 2). +/// +/// [`read::with_ranking`]: nexus_flow_facade::read::with_ranking +/// [`read::next_ranking_reads`]: nexus_flow_facade::read::next_ranking_reads +pub async fn with_ranking( + table: &T, + mut board: ActiveBoard, + reads: &RankingReads, +) -> Result> { + // The board may have been read before a refold began, and an older stream's entries say + // nothing of their edge: asked here again, one `GetItem` (review of PR #55, Integrity #3). + check_current(table).await?; + let ids: Vec = board.tickets.keys().cloned().collect(); + let held: BTreeMap = stream::iter(ids) + .map(|id| async move { + let edges = present_edges(table, &id).await?; + Ok::<_, T::Error>((id, edges)) + }) + .buffered(READ_CONCURRENCY) + .try_collect::>() + .await? + .into_iter() + .map(|(id, edges)| { + let parent = board + .tickets + .get(&id) + .and_then(|t| t.item.belongs_to.clone()); + (id, Held { parent, edges }) + }) + .collect(); + board.ranking = Some(ranking_graph(table, reads, held).await?); + Ok(board) +} + +/// A row the board holds itself — an active ticket, a listed item — as the graph reads it: its +/// current parent and the present edges touching it. +struct Held { + parent: Option, + edges: BTreeMap, +} + +/// The live, unarchived items of `item_type` on the `typed` index — the type's `list` scope — in +/// id order. +/// +/// What the index hands back is its copy of each row, and the copy's own cells are asked again: +/// the fold writes the `typed` attribute AFTER the row ([`board`](crate::board), "`typed`"), so a +/// copy taken between the two writes — or left by a run that died between them — can be retyped or +/// deleted while its attribute still names `item_type`; it is dropped. The base row itself is not +/// read again: the index's lag behind it is the one the module doc's "Two points in time" +/// describes, a state the stream really had, and is not hidden here. +async fn typed_items(table: &T, item_type: &str) -> Result, T::Error> { + Ok(table + .query_typed(item_type) + .await? + .into_iter() + .filter(|r| { + crate::board::typed(r) == Some(item_type) + && text(r, "archived").is_none() + && text(r, "id").is_some() + }) + .collect()) +} + +/// One live row the graph holds, read in full: its row with its current parent, its labels and +/// custom values when `reads` asks, and its present edges. +async fn neighbour( + table: &T, + id: &str, + row: &Row, + reads: &RankingReads, +) -> Result<(Neighbour, BTreeMap), T::Error> { + let edges = present_edges(table, id).await?; + let mut n = Neighbour::new(item_row(row, current_parent(id, &edges))); + if reads.labels { + n.labels = labels(table, id).await?; + } + if reads.custom { + n.custom = custom(table, id).await?; + } + Ok((n, edges)) +} + +/// The [`RankingGraph`] for `reads` around the rows a board holds itself (`held`, each a subject, +/// with its edges already read) — what [`with_ranking`] and [`list_board`] put on their boards. +async fn ranking_graph( + table: &T, + reads: &RankingReads, + held: BTreeMap, +) -> Result { + // The listed items of `reads.list_types` the board does not hold: subjects too. + let mut listed: BTreeSet = BTreeSet::new(); + let mut scope_rows: Vec<(String, Row)> = Vec::new(); + for ty in &reads.list_types { + for row in typed_items(table, ty).await? { + let id = text(&row, "id").unwrap_or_default().to_string(); + listed.insert(id.clone()); + if !held.contains_key(&id) { + scope_rows.push((id, row)); + } + } + } + let scope: BTreeMap)> = + stream::iter(scope_rows) + .map(|(id, row)| async move { + let read = neighbour(table, &id, &row, reads).await?; + Ok::<_, T::Error>((id, read)) + }) + .buffered(READ_CONCURRENCY) + .try_collect() + .await?; + let subject = |id: &str| held.contains_key(id) || scope.contains_key(id); + + // Every present edge of a kind read that touches a subject, once per tag. + let mut touching: BTreeMap = BTreeMap::new(); + for edges in held + .values() + .map(|h| &h.edges) + .chain(scope.values().map(|(_, e)| e)) + { + touching.extend( + edges + .iter() + .filter(|(_, e)| reads.kinds.contains(&e.kind)) + .map(|(tag, e)| (tag.clone(), e.clone())), + ); + } + + // The other ends: their rows, and for the live ones their current parent and the joins asked + // for. + let others: BTreeSet = touching + .values() + .flat_map(|e| [&e.from, &e.to]) + .filter(|id| !subject(id)) + .cloned() + .collect(); + let keys: Vec = others.iter().map(|id| item_key(id)).collect(); + let mut rows = table.get_many(&keys).await?; + let live: Vec<(String, Row)> = others + .into_iter() + .filter_map(|id| rows.remove(&item_key(&id)).map(|r| (id, r))) + .filter(|(_, r)| text(r, "deleted") != Some("1")) + .collect(); + let ends: BTreeMap = stream::iter(live) + .map(|(id, row)| async move { + let (n, _) = neighbour(table, &id, &row, reads).await?; + Ok::<_, T::Error>((id, n)) + }) + .buffered(READ_CONCURRENCY) + .try_collect() + .await?; + + // The edges between live ends, a `parent` edge only from a child to its current parent. + let parent_of = |id: &str| -> Option<&str> { + match held.get(id) { + Some(h) => h.parent.as_deref(), + None => scope + .get(id) + .map(|(n, _)| n) + .or_else(|| ends.get(id)) + .and_then(|n| n.item.belongs_to.as_deref()), + } + }; + let live_end = |id: &str| subject(id) || ends.contains_key(id); + let edges: BTreeSet = touching + .into_values() + .filter(|e| live_end(&e.from) && live_end(&e.to)) + .filter(|e| e.kind != EdgeKind::Parent || parent_of(&e.from) == Some(e.to.as_str())) + .map(|e| Edge { + from: e.from, + to: e.to, + kind: e.kind, + }) + .collect(); + // A child whose current parent is elsewhere was read for its parent edge only, which is not in + // the graph: it is no neighbour. A listed item always is one. + let edge_ends: BTreeSet<&str> = edges + .iter() + .flat_map(|e| [e.from.as_str(), e.to.as_str()]) + .collect(); + let mut neighbours: BTreeMap = ends + .into_iter() + .filter(|(id, _)| edge_ends.contains(id.as_str())) + .collect(); + neighbours.extend(scope.into_iter().map(|(id, (n, _))| (id, n))); + let mut graph = RankingGraph::new(reads.clone()); + graph.edges = edges; + graph.neighbours = neighbours; + graph.listed = listed; + Ok(graph) +} + +/// The [`ListBoard`] for `reads` from the folded tables (6j6v.xkr7) — what [`read::list_board`] +/// reads from a store, item for item — for the facade's store-free `list` +/// ([`read::list_board_page`], [`read::list_board_value`]). A server takes `reads` from +/// [`read::list_reads`] for its query. The stream must be current, as for [`active_board`]: the +/// `typed` index and the self-describing `.adj#` entries came with index revision 3. +/// +/// # What a list costs +/// +/// The listed types' rows come first: one Query on the `typed` index per listed type, or — for a +/// query without a type filter — one Query on the partition's `items#` rows (every item row of the +/// stream, deleted ones included, strongly consistent; DynamoDB pages it at 1 MB, so it is one +/// request per MB of item rows). Archived and deleted items are dropped there: `list` returns +/// neither. One `GetItem` of `.meta#watermark` per board. What follows depends on the order. +/// +/// **Any order but `rank`** ([`ListReads::select`]) reads the item rows alone, so the rows are +/// kept and cut BEFORE the joins a record carries are read ([`read::list_select`]): the status and +/// the type are asked of the row; a container filter reads one Query on `.adj#`, a label filter one +/// on `.ladj#`, per item the row kept; then only the `k` items of the page (at most `limit`) read +/// the rest: +/// +/// | read | per | +/// |--------------------------------------------------------------|---------------------------| +/// | one Query each on `.adj#`, `.ladj#`, `custom_fields#` (less what the filter read) | page item | +/// | `item_timestamps` rows, by `BatchGetItem` | page — `⌈k / 100⌉` requests | +/// | parents of open page items outside the page, by `BatchGetItem` | page — `⌈p / 100⌉` requests | +/// +/// So `list --limit 10` costs `1 + 30 + 2` requests beside the rows' Query, at most, however many +/// items the stream holds and however many the status refuses — plus one Query per item the row +/// kept when the query filters by container or label. +/// +/// **`sort = rank`** reads every live, unarchived item of the listed types in full — one Query +/// each on `.adj#`, `.ladj#`, `custom_fields#` per item, and the timestamps and outside parents by +/// `BatchGetItem` — whatever its status, filter or limit: a rank is computed over the type's whole +/// `list` scope (docs/specs/derived-ranking-keys.md §12, decision 2), and a derived key reads the +/// labels and custom values of every item in it. When a key reads a rank `next` gives, one Query on +/// the `active` index and per active ticket one Query each on `.ladj#` / `custom_fields#` when the +/// graph reads them; with derived keys, [`with_ranking`]'s reads around the listed items and the +/// active tickets. The `.adj#` Query gives each item's parent, its `contributes_to` targets and its +/// parents for `parent_closed_reason` at once, and the graph reuses it. +/// +/// Nothing here caps the requests of one call: a ranked list of a type with `n` live items costs +/// about `3n` Queries, the number its rank is computed over. A server that must bound one request +/// sets its own budget around the table it hands in (`6j6v.wq9e`, E4 spec §5.1). +/// +/// [`read::list_board`]: nexus_flow_facade::read::list_board +/// [`read::list_board_page`]: nexus_flow_facade::read::list_board_page +/// [`read::list_board_value`]: nexus_flow_facade::read::list_board_value +/// [`read::list_reads`]: nexus_flow_facade::read::list_reads +/// [`read::list_select`]: nexus_flow_facade::read::list_select +pub async fn list_board( + table: &T, + reads: &ListReads, +) -> Result> { + check_current(table).await?; + let mut rows: Vec = match &reads.types { + Some(types) => { + let mut rows = Vec::new(); + for ty in types { + rows.extend(typed_items(table, ty).await?); + } + rows + } + None => table + .query_prefix(&table_prefix("items")) + .await? + .into_iter() + .filter(|r| { + text(r, "deleted") != Some("1") + && text(r, "archived").is_none() + && text(r, "id").is_some() + }) + .collect(), + }; + let keep = reads.select.as_ref(); + // A list that is not ranked keeps what its row decides — status, type — before any join. + if let Some(select) = keep { + rows.retain(|r| select.keeps_row(&item_row(r, None))); + } + + // What keeping an item reads: for a ranked list every join below, for any other only what its + // filter reads — a container's edges, a label's labels. + let edges_first = keep.is_none_or(ListSelect::reads_edges); + let labels_first = keep.is_none_or(ListSelect::reads_labels); + type Read = (Row, Option>, ListItem); + let first: Vec = stream::iter(rows) + .map(|row| async move { + let id = text(&row, "id").unwrap_or_default().to_string(); + let edges = if edges_first { + Some(present_edges(table, &id).await?) + } else { + None + }; + let mut i = list_item(&row, &id, edges.as_ref()); + if labels_first { + i.labels = labels(table, &id).await?; + } + Ok::<_, T::Error>((row, edges, i)) + }) + .buffered(READ_CONCURRENCY) + .try_collect() + .await?; + + // Kept and cut as `list_board_page` keeps and cuts, so the joins below are read for the page. + let (first, cut) = match keep { + None => (first, None), + Some(select) => { + let mut held: BTreeMap>)> = + BTreeMap::new(); + let items: Vec = first + .into_iter() + .map(|(row, edges, i)| { + held.insert(i.item.id.clone(), (row, edges)); + i + }) + .collect(); + let (kept, cut) = read::list_select(select, items); + let kept = kept + .into_iter() + .filter_map(|i| held.remove(&i.item.id).map(|(row, edges)| (row, edges, i))) + .collect(); + (kept, Some(cut)) + } + }; + + // The rest of each kept item's joins. + let mut read: Vec<(Row, BTreeMap, ListItem)> = stream::iter(first) + .map(|(row, edges, mut i)| async move { + let id = i.item.id.clone(); + let edges = match edges { + Some(edges) => edges, + None => { + let edges = present_edges(table, &id).await?; + let full = list_item(&row, &id, Some(&edges)); + i.item = full.item; + i.contributes_to = full.contributes_to; + edges + } + }; + if !labels_first { + i.labels = labels(table, &id).await?; + } + i.custom = custom(table, &id).await?; + Ok::<_, T::Error>((row, edges, i)) + }) + .buffered(READ_CONCURRENCY) + .try_collect() + .await?; + let keys: Vec = read + .iter() + .map(|(_, _, i)| timestamps_key(&i.item.id)) + .collect(); + let stamped = table.get_many(&keys).await?; + for (_, _, i) in &mut read { + (i.created_at, i.updated_at) = stamps(stamped.get(×tamps_key(&i.item.id))); + } + + // `parent_closed_reason`: an OPEN item with parents, none of them active, lists them all. A + // parent outside the list is read once, however many children it has. + let parents_of = |id: &str, edges: &BTreeMap| -> BTreeSet { + edges + .values() + .filter(|e| e.kind == EdgeKind::Parent && e.from == id) + .map(|e| e.to.clone()) + .collect() + }; + let listed_rows: BTreeMap<&str, &Row> = read + .iter() + .map(|(row, _, i)| (i.item.id.as_str(), row)) + .collect(); + let outside: Vec = read + .iter() + .filter(|(_, _, i)| i.item.status.as_deref() == Some("open")) + .flat_map(|(_, edges, i)| parents_of(&i.item.id, edges)) + .filter(|p| !listed_rows.contains_key(p.as_str())) + .map(|p| item_key(&p)) + .collect(); + let outside = table.get_many(&outside).await?; + let parent_row = |pid: &str| -> Option<&Row> { + listed_rows + .get(pid) + .copied() + .or_else(|| outside.get(&item_key(pid))) + }; + let parent_closed: Vec)>> = read + .iter() + .map(|(_, edges, i)| { + if i.item.status.as_deref() != Some("open") { + return Vec::new(); + } + let mut closed = Vec::new(); + for pid in parents_of(&i.item.id, edges) { + match parent_row(&pid) { + Some(r) if is_active(r) => return Vec::new(), + Some(r) => closed.push(( + text(r, "id").unwrap_or(&pid).to_string(), + text(r, "closing_comment").map(str::to_string), + )), + None => closed.push((pid, None)), + } + } + closed.sort_by(|a, b| a.0.cmp(&b.0)); + closed + }) + .collect(); + let mut items = Vec::with_capacity(read.len()); + let mut held: BTreeMap = BTreeMap::new(); + for ((_, edges, mut i), closed) in read.into_iter().zip(parent_closed) { + i.parent_closed = closed; + held.insert( + i.item.id.clone(), + Held { + parent: i.item.belongs_to.clone(), + edges, + }, + ); + items.push(i); + } + let mut board = ListBoard::new(reads.types.clone(), items); + board.cut = cut; + + let graph_reads = reads.ranking.clone().unwrap_or_default(); + if reads.next { + let Selection { + items: rows, + board: lanes, + parents, + } = select(table).await?; + let tickets: Vec = stream::iter(rows) + .map(|(id, row)| { + let parent = parents.get(&id).cloned(); + let graph_reads = &graph_reads; + async move { + let mut t = ActiveTicket::new(item_row(&row, parent)); + if graph_reads.labels { + t.labels = labels(table, &id).await?; + } + if graph_reads.custom { + t.custom = custom(table, &id).await?; + } + Ok::<_, T::Error>(t) + } + }) + .buffered(READ_CONCURRENCY) + .try_collect() + .await?; + board.active = Some(ActiveBoard::new(lanes, tickets)); + } + if let Some(r) = &reads.ranking { + if let Some(active) = &board.active { + let missing: Vec<(String, Option)> = active + .tickets + .iter() + .filter(|(id, _)| !held.contains_key(*id)) + .map(|(id, t)| (id.clone(), t.item.belongs_to.clone())) + .collect(); + let more: Vec<(String, Held)> = stream::iter(missing) + .map(|(id, parent)| async move { + let edges = present_edges(table, &id).await?; + Ok::<_, T::Error>((id, Held { parent, edges })) + }) + .buffered(READ_CONCURRENCY) + .try_collect() + .await?; + held.extend(more); + } + board.ranking = Some(ranking_graph(table, r, held).await?); + } + Ok(board) +} + +/// A listed item from its row and, when they were read, its present edges: its `belongs_to` and its +/// `contributes_to` targets come from them, and stay empty without them. +fn list_item(row: &Row, id: &str, edges: Option<&BTreeMap>) -> ListItem { + let mut i = ListItem::new(item_row(row, edges.and_then(|e| current_parent(id, e)))); + if let Some(edges) = edges { + let targets: BTreeSet = edges + .values() + .filter(|e| e.kind == EdgeKind::ContributesTo && e.from == id) + .map(|e| e.to.clone()) + .collect(); + i.contributes_to = targets.into_iter().collect(); + } + i +} + +/// Refuse a stream folded by older index rules than this build reads, and one being folded again +/// that has not caught up yet. A cleared stream's watermark names no revision +/// ([`Folder::clear`]), so it is refused from the clear on; once the refold has written this +/// build's revisions it is still refused until it reaches the position the clear named. A stream +/// with no watermark at all has never been folded: it reads as the empty board it is. +async fn check_current(table: &T) -> Result<(), RecordsError> { + if let Some(mark) = Folder::watermark(table).await? { + let found = mark.revisions.get(INDEX_DOMAIN).copied(); + if found != Some(INDEX_REVISION) { + return Err(RecordsError::NotCurrent { found }); + } + if let Some(until) = mark.refold_until.filter(|u| mark.folded_through < *u) { + return Err(RecordsError::Refolding { + folded_through: mark.folded_through, + until, + }); + } + } + Ok(()) +} + /// An `items` row as the store reads it, with the ticket's `belongs_to` (the `present_parent` join). pub(crate) fn item_row(row: &Row, belongs_to: Option) -> ItemRow { let cell = |c: &str| text(row, c).map(str::to_string); @@ -243,16 +851,23 @@ async fn timestamps( table: &T, id: &str, ) -> Result<(Option, Option), T::Error> { - let row = table - .get(&key("item_timestamps", &[("item_id", id)])) - .await?; + let row = table.get(×tamps_key(id)).await?; + Ok(stamps(row.as_ref())) +} + +/// The sort key of the ticket's `item_timestamps` row. +fn timestamps_key(id: &str) -> String { + key("item_timestamps", &[("item_id", id)]) +} + +/// The creation and last change an `item_timestamps` row holds, `''` read as none. +fn stamps(row: Option<&Row>) -> (Option, Option) { let stamp = |c: &str| { - row.as_ref() - .and_then(|r| text(r, c)) + row.and_then(|r| text(r, c)) .filter(|v| !v.is_empty()) .map(str::to_string) }; - Ok((stamp("created_at"), stamp("updated_at"))) + (stamp("created_at"), stamp("updated_at")) } /// Every non-empty custom value of the ticket, by field. diff --git a/crates/fold-ddb/src/snapshot.rs b/crates/fold-ddb/src/snapshot.rs index 55b90a7..e3c9091 100644 --- a/crates/fold-ddb/src/snapshot.rs +++ b/crates/fold-ddb/src/snapshot.rs @@ -7,9 +7,11 @@ //! change is idempotent. The order is what makes that safe: tables read before the watermark could //! lack an op the watermark claims. //! -//! [`import`] starts an EMPTY stream from one: it writes the rows, rebuilds the board's indexes from -//! them, and writes the watermark last, so a stream whose import died has no watermark and is -//! started again. The log stays where it is — on the relay — and the run then folds it from the +//! [`import`] starts an EMPTY stream from one — one never folded, or one [`Folder::clear`] left — +//! and keeps it unserved while it does: it writes a sentinel watermark that names no revision +//! first, so readers answer `NotCurrent` throughout; then the rows; it rebuilds the board's +//! indexes from them, and writes the real watermark last, so a stream whose import died is still +//! unserved and is started again. The log stays where it is — on the relay — and the run then folds it from the //! watermark on, in batches. //! //! [`from_sqlite`] makes one from a local replica's views, for a server that starts from a board @@ -186,10 +188,12 @@ pub async fn import( folder: folder.revisions(), }); } + // A cleared stream's watermark — and that of an import that died — names no revision: it + // holds nothing to keep. if Folder::watermark(table) .await .map_err(ImportError::Table)? - .is_some() + .is_some_and(|mark| !mark.revisions.is_empty()) { return Err(ImportError::NotEmpty); } @@ -232,18 +236,23 @@ pub async fn import( }); } } + // Unserved from the first row on: a watermark without revisions (review of PR #55, + // Integrity #2). + table + .write(&Folder::sentinel(0, None)) + .await + .map_err(ImportError::Table)?; for write in &writes { table.write(write).await.map_err(ImportError::Table)?; } + // An item whose row has no room for its index attributes keeps none: as with the row writes + // above, whose `TooLarge` an import does not report either (`6j6v.jtfy`). board::reindex(table).await.map_err(ImportError::Table)?; - // A NEW watermark, not an advance: the stream had none. - folder - .advance(table, snapshot.folded_through) + // The real watermark over the sentinel, unconditionally: the stream had none of its own. + table + .write(&folder.stamp(snapshot.folded_through)) .await - .map_err(|e| match e { - crate::fold::FoldError::Table(e) => ImportError::Table(e), - crate::fold::FoldError::Layout(e) => ImportError::Layout(e), - })?; + .map_err(ImportError::Table)?; Ok(()) } @@ -313,5 +322,6 @@ pub fn watermark(snapshot: &Snapshot) -> Watermark { Watermark { folded_through: snapshot.folded_through, revisions: snapshot.revisions.clone(), + refold_until: None, } } diff --git a/crates/fold-ddb/src/table.rs b/crates/fold-ddb/src/table.rs index c382c1e..b60ffab 100644 --- a/crates/fold-ddb/src/table.rs +++ b/crates/fold-ddb/src/table.rs @@ -1,6 +1,6 @@ -//! The five requests a server fold makes of its table, as a trait: [`MemTable`](crate::mem::MemTable) +//! The requests a server fold makes of its table, as a trait: [`MemTable`](crate::mem::MemTable) //! answers them in memory for the tests, `ddb::DynamoDbTable` (feature `dynamodb`) answers them with DynamoDB. -//! None of them is a Scan — every read names its partition, and the two board reads name an index. +//! None of them is a Scan — every read names its partition, and the three index reads name an index. use crate::write::Write; use nxs_foundation::change::Cell; @@ -10,6 +10,9 @@ use std::future::Future; /// A stored row: its attributes by name, the sort key and index attributes among them. pub type Row = BTreeMap; +/// The most keys one `BatchGetItem` names — DynamoDB's own limit ([`Table::get_many`]). +pub const BATCH_GET_KEYS: usize = 100; + /// What a write did. #[derive(Debug, Clone, Copy, PartialEq, Eq)] pub enum Outcome { @@ -49,6 +52,16 @@ pub trait Table: Sync { /// One row, read strongly consistent. fn get(&self, sk: &str) -> impl Future, Self::Error>> + Send; + /// Several rows, read strongly consistent, by sort key — a key without a row is absent from the + /// answer, and a key named twice is read once. DynamoDB answers it with `BatchGetItem`, at most + /// [`BATCH_GET_KEYS`] keys a request, sending the keys a request left unprocessed again; so it + /// costs `⌈keys / 100⌉` requests, more only when DynamoDB throttles or a response passes 16 MB. + /// No keys, no request. + fn get_many( + &self, + sks: &[String], + ) -> impl Future, Self::Error>> + Send; + /// One conditional write. fn write(&self, write: &Write) -> impl Future> + Send; @@ -71,6 +84,13 @@ pub trait Table: Sync { lane: Dated, limit: Option, ) -> impl Future, Self::Error>> + Send; + + /// Every row on the `typed` index under `item_type` (6j6v.xkr7): the live items of that type, + /// archived ones included, in sort-key (id) order. Eventually consistent, as every index read. + fn query_typed( + &self, + item_type: &str, + ) -> impl Future, Self::Error>> + Send; } /// A text cell of `row`, `None` when it is absent or NULL. diff --git a/crates/fold-ddb/tests/common/mod.rs b/crates/fold-ddb/tests/common/mod.rs index eeffa36..55094dc 100644 --- a/crates/fold-ddb/tests/common/mod.rs +++ b/crates/fold-ddb/tests/common/mod.rs @@ -25,6 +25,10 @@ impl Lcg { } } +/// The item types of a generated board: items are created as one of the first two and retyped +/// among all three, so the `typed` index sees a type gained, lost and changed (6j6v.xkr7). +pub const TYPES: [&str; 3] = ["task", "epic", "note"]; + pub fn id(i: u64) -> String { format!("ab12.{i:04}") } @@ -38,6 +42,11 @@ fn write(s: &mut Store, r: &mut Lcg, n: u64, steps: usize) { step % 24 )); let (a, b) = (id(r.next(n)), id(r.next(n + 1))); + // Now and then a retype (6j6v.xkr7), on the step alone so the random stream — and with it + // every other op of a seed — stays as it was. + if step % 9 == 4 { + s.set_field(&a, "type", Some(TYPES[step % TYPES.len()].into()), "u"); + } match r.next(16) { 0 => s.create_item(&a, "task", "T", "u"), 1 => s.set_field(&a, "status", Some("in_progress".into()), "u"), @@ -149,7 +158,7 @@ pub fn merged(seed: u64) -> Store { let mut a = Store::open_in_memory(1); let mut b = Store::open_in_memory(2); for i in 0..n { - a.create_item(&id(i), "task", "T", "u"); + a.create_item(&id(i), TYPES[(i % 2) as usize], "T", "u"); } b.apply(&a.export()); let steps = 10 + r.next(40) as usize; @@ -176,6 +185,12 @@ pub fn delivered(s: &Store, seed: u64) -> Vec<(i64, Op)> { .collect() } +/// The highest relay position among `ops` — the head a refold of them has to reach +/// (`Folder::clear`). +pub fn head(ops: &[(i64, Op)]) -> i64 { + ops.iter().map(|(at, _)| *at).max().unwrap_or(0) +} + /// Every row of `table`, every column filled — as SQLite holds it. pub fn sqlite_rows(s: &Store, table: &str) -> Vec> { let conn = s.connection(); diff --git a/crates/fold-ddb/tests/derived_differential.rs b/crates/fold-ddb/tests/derived_differential.rs new file mode 100644 index 0000000..8beb9c9 --- /dev/null +++ b/crates/fold-ddb/tests/derived_differential.rs @@ -0,0 +1,528 @@ +//! A server ranks `next` by derived ranking keys as a replica does (6j6v.1c5s, +//! docs/specs/derived-ranking-keys.md §6 and §9 "differential"). +//! +//! Generated boards of three types — goals, projects, actions — written by two replicas apart and +//! then merged: every status, archived and deleted items, priorities, the goals' `order` custom +//! field, `waiting` labels, and `parent`, `contributes_to`, `mentions` and `dep` edges with removes. +//! Two replicas parent the same child apart, so a child can have two present parent edges and only +//! `present_parent`'s pick may count. Each board is folded into a [`MemTable`] in a shuffled order +//! with duplicates, as the relay hands it over. +//! +//! Then three paths rank `next` under a plugin whose orders hold every derived key kind, each +//! reading closed, archived and deleted neighbours, down a three-type chain (action → project → +//! goal): +//! +//! 1. the SQL path: `read::next` over the store; +//! 2. the Board path: `read::next_active` over the store's own [`read::ActiveBoard`] and +//! [`read::RankingGraph`] (`read::active_board` + `read::with_ranking`); +//! 3. the fold-ddb path: `read::next_active` over the folded tables' (`board::select` + +//! `records::active_board` + `records::with_ranking`). +//! +//! All three must give the same rows in the same order and the same JSON, and the two inputs must +//! be equal, edge for edge and neighbour for neighbour. Folding the same log in another order must +//! read the same graph (determinism). + +use nexus_flow_core::model::{EdgeKind, ItemRow, MergeStrategy}; +use nexus_flow_core::store::Store; +use nexus_flow_facade::plugin::{self, PluginConfig, PluginRegistration}; +use nexus_flow_facade::read; +use nxs_fold_ddb::fold::{Folder, INDEX_DOMAIN, INDEX_REVISION}; +use nxs_fold_ddb::layout::WATERMARK_KEY; +use nxs_fold_ddb::mem::{ready, MemTable}; +use nxs_fold_ddb::table::Table; +use nxs_fold_ddb::write::{Cond, Write}; +use nxs_fold_ddb::{board, records}; +use nxs_foundation::change::Cell; +use std::collections::{BTreeMap, BTreeSet}; + +mod common; +use common::{delivered, head, Lcg, NOW}; + +/// Every derived key kind, each able to read a neighbour that is not active: a goal's order and +/// rank from a closed or archived goal, counts without a status filter, a `dep` blocker's status +/// (only a closed blocker leaves its dependent in `next`), the `belongs_to` of a mentioned item. +const ORDERS: &str = r#" +name = "derived-differential-fixture" +[description] +en = "x" +de = "y" +[priority] +labels = ["P0", "P1", "P2", "P3", "P4"] +[types] +list = ["goal", "project", "action"] +[vocabulary.status] +open = "open" +in_progress = "in progress" +closed = "closed" +[ranking.next] +types = ["action", "project", "goal"] +order = [ { field = "priority", dir = "asc" } ] +[ranking.next.goal] +order = [ + { count = "contributors", type = "project", dir = "desc" }, + { field = "priority", dir = "asc" }, +] +[ranking.next.project] +order = [ + { related = "contributes_to", type = "goal", of = { field = "order" }, pick = "min", nulls = "last" }, + { related = "parent", type = "goal", of = "rank", pick = "max", nulls = "first" }, + { sum = [ { count = "mentioned_by" }, + { count = "mentioned_by", type = "action", label = "waiting", status = ["open", "in_progress"] } ], dir = "desc" }, + { count = "children", type = "action", dir = "desc" }, + { field = "priority", dir = "asc" }, +] +[ranking.next.action] +order = [ + { related = "parent", type = "project", of = "rank", nulls = "last" }, + { related = "dep", of = { field = "status", precedence = ["closed", "in_progress", "open"] }, pick = "min" }, + { count = "dependents", dir = "desc" }, + { related = "mentions", of = { field = "belongs_to" }, pick = "max", nulls = "last" }, + { count = "contributors", status = ["closed"], dir = "desc" }, + { field = "priority", dir = "asc" }, +] +[presentation.list] +columns = ["id"] +[fields.order] +type = "text" +on = ["goal"] +"#; + +inventory::submit!(PluginRegistration { + name: "derived-differential-fixture", + toml: ORDERS, + order: 140, +}); + +fn cfg() -> PluginConfig { + plugin::load("derived-differential-fixture").expect("the fixture is a valid plugin") +} + +/// The items of a generated board: three goals, five projects, eight actions. +fn items() -> Vec<(String, &'static str)> { + let mut out = Vec::new(); + for (ty, letter, n) in [("goal", 'g', 3), ("project", 'p', 5), ("action", 'a', 8)] { + for i in 0..n { + out.push((format!("ab12.{letter}{i:03}"), ty)); + } + } + out +} + +/// One replica's share of the writing. `added` holds the edges this replica added, so a remove +/// usually takes back a real one. +fn write(s: &mut Store, r: &mut Lcg, added: &mut Vec<(String, String, EdgeKind)>, steps: usize) { + let all = items(); + let pick = |r: &mut Lcg| all[r.next(all.len() as u64) as usize].0.clone(); + for step in 0..steps { + s.set_wall_clock(&format!( + "2026-10-0{}T{:02}:00:00Z", + 1 + r.next(7), + step % 24 + )); + let (x, y) = (pick(r), pick(r)); + match r.next(16) { + 0 => s.set_field(&x, "status", Some("in_progress".into()), "u"), + 1 | 2 => s.set_field(&x, "status", Some("closed".into()), "u"), + 3 => s.set_field(&x, "status", Some("open".into()), "u"), + 4 => s.set_field( + &x, + "archived", + Some(format!("2026-10-0{}", 1 + r.next(7))), + "u", + ), + 5 => s.set_field(&x, "archived", None, "u"), + 6 if r.next(3) == 0 => s.delete_item(&x, "u"), + 7 => s.set_field(&x, "priority", Some(r.next(5).to_string()), "u"), + 8..=12 => { + // Dependencies are rare: an active blocker takes its dependent out of `next`. + let kind = match r.next(10) { + 0..=2 => EdgeKind::Parent, + 3..=5 => EdgeKind::ContributesTo, + 6..=8 => EdgeKind::Mentions, + _ => EdgeKind::Dep, + }; + s.add_edge(&x, &y, kind, "u"); + added.push((x, y, kind)); + } + 13 if !added.is_empty() => { + let (from, to, kind) = added.swap_remove(r.next(added.len() as u64) as usize); + s.remove_edge(&from, &to, kind, "u"); + } + 14 => { + if r.next(3) == 0 { + s.remove_label(&x, "waiting", "u"); + } else { + s.add_label(&x, "waiting", "u"); + } + } + _ => s.set_custom_field_merge( + &x, + "order", + Some(r.next(6).to_string()), + "u", + MergeStrategy::Lww, + ), + } + } +} + +/// A merged board: two replicas that wrote apart, then exchanged everything. +fn gen_board(seed: u64) -> Store { + let mut r = Lcg(seed ^ 0xd1ff); + let mut a = Store::open_in_memory(1); + for (id, ty) in items() { + a.create_item(&id, ty, "T", "u"); + a.set_field(&id, "priority", Some(r.next(5).to_string()), "u"); + if ty == "goal" && r.next(4) != 0 { + a.set_custom_field_merge( + &id, + "order", + Some(r.next(6).to_string()), + "u", + MergeStrategy::Lww, + ); + } + } + let mut b = Store::open_in_memory(2); + b.apply(&a.export()); + let steps = 30 + r.next(40) as usize; + write(&mut a, &mut r, &mut Vec::new(), steps); + write(&mut b, &mut r, &mut Vec::new(), steps); + let from_b = b.export(); + a.apply(&from_b); + a +} + +/// `ops` folded into a fresh table. +fn fold(folder: &Folder, ops: &[(i64, nxs_foundation::model::Op)]) -> MemTable { + let t = MemTable::new("stream-1"); + ready(folder.fold_batch(&t, ops)).unwrap(); + t +} + +/// The served board with its ranking graph, as a server reads it for `reads` — and with the +/// contributes-to targets, so its tickets compare whole against `read::active_board`'s. +fn served(t: &MemTable, reads: &read::RankingReads) -> read::ActiveBoard { + let selection = ready(board::select(t)).unwrap(); + let plain = ready(records::active_board(t, selection)).unwrap(); + let plain = ready(records::with_contributes_to(t, plain)).unwrap(); + ready(records::with_ranking(t, plain, reads)).unwrap() +} + +fn ids(items: &[ItemRow]) -> Vec { + items.iter().map(|i| i.id.clone()).collect() +} + +/// An active ticket, as the SQL `items` row `a` says: live, unarchived, open or in progress. +const ACTIVE_SQL: &str = "COALESCE(a.deleted, '') != '1' AND a.archived IS NULL + AND a.status IN ('open', 'in_progress')"; + +fn count(s: &Store, sql: &str) -> usize { + s.connection() + .query_row(sql, [], |r| r.get::<_, i64>(0)) + .unwrap() as usize +} + +/// What the generated boards held, so the comparison is known to have compared something worth +/// comparing. +#[derive(Default)] +struct Seen(BTreeMap<&'static str, usize>); + +impl Seen { + fn add(&mut self, what: &'static str, n: usize) { + *self.0.entry(what).or_default() += n; + } + + fn assert_all(&self) { + for (what, n) in &self.0 { + assert!(*n > 0, "the generated boards have {what}: {:?}", self.0); + } + } +} + +#[test] +fn the_server_ranks_next_by_derived_keys_as_the_replica_does() { + let folder = Folder::platform().unwrap(); + let cfg = cfg(); + let reads = read::next_ranking_reads(&cfg) + .unwrap() + .expect("derived next keys"); + assert_eq!(reads.kinds.len(), 4, "every edge kind: {reads:?}"); + assert!(reads.labels && reads.custom, "{reads:?}"); + let mut seen = Seen::default(); + for seed in 0..60 { + let s = gen_board(seed); + let ctx = format!("seed {seed}"); + let t = fold(&folder, &delivered(&s, seed)); + + let local = read::with_ranking(&s, read::active_board(&s).unwrap(), &reads).unwrap(); + let server = served(&t, &reads); + assert_eq!(server.tickets, local.tickets, "active tickets, {ctx}"); + assert_eq!(server.ranking, local.ranking, "the ranking graph, {ctx}"); + + // 1 = 2 = 3: the SQL path, the Board path, the fold-ddb path. + let want = read::next(&cfg, &s, NOW, None).unwrap(); + assert_eq!( + ids(&read::next_active(&cfg, &local, NOW).unwrap()), + ids(&want), + "SQL path vs Board path, {ctx}" + ); + assert_eq!( + ids(&read::next_active(&cfg, &server, NOW).unwrap()), + ids(&want), + "SQL path vs fold-ddb path, {ctx}" + ); + assert_eq!( + read::next_active_value(&cfg, &server, NOW).unwrap(), + read::next_to_value_with_custom(&cfg, &s, &want).unwrap(), + "next JSON, {ctx}" + ); + + // Determinism: the same log folded in another order reads the same graph (every fourth + // board — a fold is most of a seed's time). + if seed % 4 == 0 { + let again = served(&fold(&folder, &delivered(&s, seed ^ 0xffff)), &reads); + assert_eq!(again.ranking, server.ranking, "another fold order, {ctx}"); + } + + // What the boards held. + let graph = server.ranking.as_ref().unwrap(); + let candidates: BTreeSet<&str> = want.iter().map(|i| i.id.as_str()).collect(); + seen.add("next rows", want.len()); + for n in graph.neighbours.values() { + let archived = n.item.archived.is_some(); + seen.add( + "closed neighbours", + usize::from(!archived && n.item.status.as_deref() == Some("closed")), + ); + seen.add("archived neighbours", usize::from(archived)); + seen.add("labelled neighbours", usize::from(!n.labels.is_empty())); + seen.add( + "goal neighbours with an order", + usize::from(n.custom.contains_key("order")), + ); + seen.add( + "neighbours with a parent", + usize::from(n.item.belongs_to.is_some()), + ); + } + for e in &graph.edges { + let into_candidate = candidates.contains(e.to.as_str()); + let from_neighbour = graph.neighbours.contains_key(&e.from); + seen.add( + "inbound contributes_to to a candidate", + usize::from(into_candidate && e.kind == EdgeKind::ContributesTo), + ); + seen.add( + "mentions of a candidate from an inactive item", + usize::from(into_candidate && from_neighbour && e.kind == EdgeKind::Mentions), + ); + seen.add( + "inactive children of a candidate", + usize::from(into_candidate && from_neighbour && e.kind == EdgeKind::Parent), + ); + seen.add( + "dep edges from a candidate to an inactive blocker", + usize::from( + candidates.contains(e.from.as_str()) + && graph.neighbours.contains_key(&e.to) + && e.kind == EdgeKind::Dep, + ), + ); + } + // Counted where they matter (review of PR #55, Test #4): at an active ticket, whose edges + // the graph reads — not anywhere on the board. + seen.add( + "edges dropped for a deleted end", + count( + &s, + &format!( + "SELECT COUNT(*) FROM present_edges e + JOIN items d ON d.id IN (e.from_id, e.to_id) AND d.deleted = '1' + JOIN items a ON a.id IN (e.from_id, e.to_id) AND a.id != d.id + WHERE {ACTIVE_SQL}" + ), + ), + ); + // A child with two present parent edges: only `present_parent`'s pick may count. + seen.add( + "children parented twice", + count( + &s, + "SELECT COUNT(*) FROM (SELECT from_id FROM present_edges WHERE kind = 'parent' + GROUP BY from_id HAVING COUNT(*) > 1)", + ), + ); + seen.add( + "removed edges at an active ticket", + count( + &s, + &format!( + "SELECT COUNT(*) FROM edge_removes r JOIN edge_adds e ON e.tag = r.tag + JOIN items a ON a.id IN (e.from_id, e.to_id) WHERE {ACTIVE_SQL}" + ), + ), + ); + // A derived key decides: a project ranked ahead of one with a better priority. + let projects: Vec<&ItemRow> = want + .iter() + .filter(|i| i.item_type.as_deref() == Some("project")) + .collect(); + seen.add( + "projects ordered against their priority", + projects + .windows(2) + .filter(|w| w[0].priority > w[1].priority) + .count(), + ); + } + seen.assert_all(); + for what in [ + "closed neighbours", + "archived neighbours", + "inbound contributes_to to a candidate", + "mentions of a candidate from an inactive item", + "inactive children of a candidate", + "dep edges from a candidate to an inactive blocker", + "edges dropped for a deleted end", + "removed edges at an active ticket", + "children parented twice", + "projects ordered against their priority", + ] { + assert!(seen.0.contains_key(what), "{what} is counted"); + } +} + +#[test] +fn reading_the_ranking_graph_costs_one_query_per_ticket_and_per_live_neighbour() { + // §9 "bounded cost": every edge touching an active ticket is read once, from its entries; + // each inactive neighbour costs one GetItem and, when live, one Query for its parent plus one + // per join asked for — and nothing grows with history: an edge added and removed many times + // is still one entry per add, read in the same Query. + let folder = Folder::platform().unwrap(); + let cfg = cfg(); + let reads = read::next_ranking_reads(&cfg).unwrap().unwrap(); + let mut s = Store::open_in_memory(1); + s.create_item("ab12.g000", "goal", "Closed goal", "u"); + s.set_field("ab12.g000", "status", Some("closed".into()), "u"); + s.create_item("ab12.g001", "goal", "Deleted goal", "u"); + let n = 10u64; + for i in 0..n { + let id = format!("ab12.p{i:03}"); + s.create_item(&id, "project", "P", "u"); + for round in 0..5 { + s.add_edge(&id, "ab12.g000", EdgeKind::ContributesTo, "u"); + if round < 4 { + s.remove_edge(&id, "ab12.g000", EdgeKind::ContributesTo, "u"); + } + } + s.add_edge(&id, "ab12.g001", EdgeKind::Mentions, "u"); + } + s.delete_item("ab12.g001", "u"); + let t = fold(&folder, &delivered(&s, 1)); + let plain = ready(records::active_board(&t, ready(board::select(&t)).unwrap())).unwrap(); + let before = t.requests(); + let server = ready(records::with_ranking(&t, plain, &reads)).unwrap(); + let after = t.requests(); + // One adjacency Query per active ticket; the live neighbour's parent, labels and custom values. + assert_eq!(after.queries - before.queries, n + 3); + // The watermark, asked again (review of PR #55, Integrity #3); the rows of the inactive + // neighbours — the closed goal and the deleted one — in one `BatchGetItem`. + assert_eq!(after.gets - before.gets, 1); + assert_eq!(after.batch_gets - before.batch_gets, 1); + assert_eq!(after.index_queries, before.index_queries); + assert_eq!(after.writes, before.writes); + let local = read::with_ranking(&s, read::active_board(&s).unwrap(), &reads).unwrap(); + assert_eq!(server.ranking, local.ranking); + let graph = server.ranking.unwrap(); + assert_eq!( + graph.edges.len(), + n as usize, + "one contributes_to per project" + ); + assert_eq!( + graph.neighbours.keys().collect::>(), + ["ab12.g000"], + "the closed goal is a neighbour, the deleted one is not" + ); +} + +#[test] +fn a_stream_folded_by_index_revision_2_is_refused_then_refolded_and_ranks() { + // A stream folded before the adjacency entries said what their edge is (index revision 2): its + // entries hold only the tag, which would read no edge at all — so it is refused, and the fold + // run's rule refolds it. + let folder = Folder::platform().unwrap(); + let cfg = cfg(); + let reads = read::next_ranking_reads(&cfg).unwrap().unwrap(); + for seed in [3, 11, 27] { + let s = gen_board(seed); + let ops = delivered(&s, seed); + let t = fold(&folder, &ops); + // A board read while the stream was current — a server that holds one across the refold. + let plain = ready(records::active_board(&t, ready(board::select(&t)).unwrap())).unwrap(); + for (sk, row) in t.rows() { + if let Some(Cell::Text(tag)) = row.get("tag").filter(|_| sk.starts_with(".adj#")) { + ready( + t.write(&Write { + sk: sk.clone(), + set: vec![("tag".into(), Cell::Text(tag.clone()))], + remove: ["from_id", "to_id", "kind", "lamport", "site", "removed"] + .map(String::from) + .to_vec(), + condition: Some(Cond::RowPresent), + }), + ) + .unwrap(); + } + } + ready(t.write(&Write { + sk: WATERMARK_KEY.into(), + set: vec![(format!("revision_{INDEX_DOMAIN}"), Cell::Int(2))], + remove: Vec::new(), + condition: Some(Cond::RowPresent), + })) + .unwrap(); + + let selection = ready(board::select(&t)).unwrap(); + match ready(records::active_board(&t, selection)) { + Err(records::RecordsError::NotCurrent { found: Some(2) }) => {} + other => panic!("seed {seed}: a revision-2 stream must be refused, got {other:?}"), + } + // The graph read refuses it itself (review of PR #55, Integrity #3)… + match ready(records::with_ranking(&t, plain.clone(), &reads)) { + Err(records::RecordsError::NotCurrent { found: Some(2) }) => {} + other => panic!( + "seed {seed}: the graph of a revision-2 stream must be refused, got {other:?}" + ), + } + // …because the stripped entries would read as no edge at all: under a watermark that + // claims revision 3, the same read answers an empty graph where the store has edges. + let local = read::with_ranking(&s, read::active_board(&s).unwrap(), &reads).unwrap(); + let claim = |revision: i64| { + ready(t.write(&Write { + sk: WATERMARK_KEY.into(), + set: vec![(format!("revision_{INDEX_DOMAIN}"), Cell::Int(revision))], + remove: Vec::new(), + condition: Some(Cond::RowPresent), + })) + .unwrap(); + }; + claim(INDEX_REVISION); + let stripped = ready(records::with_ranking(&t, plain.clone(), &reads)).unwrap(); + assert!(stripped.ranking.unwrap().edges.is_empty(), "seed {seed}"); + assert!(!local.ranking.unwrap().edges.is_empty(), "seed {seed}"); + claim(2); + let mark = ready(Folder::watermark(&t)).unwrap().unwrap(); + assert!(!folder.is_current(&mark), "seed {seed}"); + ready(Folder::clear(&t, head(&ops))).unwrap(); + ready(folder.fold_batch(&t, &ops)).unwrap(); + + let server = served(&t, &reads); + assert_eq!( + ids(&read::next_active(&cfg, &server, NOW).unwrap()), + ids(&read::next(&cfg, &s, NOW, None).unwrap()), + "seed {seed}" + ); + } +} diff --git a/crates/fold-ddb/tests/dynamodb_local.rs b/crates/fold-ddb/tests/dynamodb_local.rs index 7c49649..7292c4f 100644 --- a/crates/fold-ddb/tests/dynamodb_local.rs +++ b/crates/fold-ddb/tests/dynamodb_local.rs @@ -1,6 +1,8 @@ //! The DynamoDB adapter against a real DynamoDB Local (6j6v.k7w7): the same fold written into //! DynamoDB and into a [`MemTable`] leaves the same items, the board reads back through the indexes -//! with the same lanes, and every request the client sends is recorded — none is a Scan. +//! with the same lanes and the same `typed` listings (6j6v.xkr7), the ranked list and the ranking +//! graph read back as from memory — a hub whose adjacency spans Query pages among them (review of +//! PR #55) — and every request the client sends is recorded — none is a Scan. //! //! Gated like `nxs-server`'s suite: on the `dynamodb` feature and `NXF_RELAY_DDB_ENDPOINT`, skipped //! locally without one and a hard failure under CI. Each test creates its own table from @@ -16,7 +18,7 @@ use aws_sdk_dynamodb::config::{ use aws_sdk_dynamodb::error::BoxError; use aws_sdk_dynamodb::types::TableStatus; use aws_sdk_dynamodb::Client; -use common::{by_key, delivered, merged, NOW}; +use common::{by_key, delivered, merged, NOW, TYPES}; use nexus_flow_core::model::EdgeKind; use nexus_flow_core::store::Store; use nxs_fold_ddb::ddb::{create_table, DynamoDbTable}; @@ -181,6 +183,11 @@ async fn a_board_folded_into_dynamodb_reads_back_as_from_memory_and_no_request_i counted.queries + counted.index_queries, "{sent:?}" ); + assert_eq!( + sent.get("BatchGetItem").copied().unwrap_or(0), + counted.batch_gets, + "{sent:?}" + ); let want = board::select(&mem).await.unwrap().board.lanes(NOW); let ddb_ref = &ddb; @@ -204,6 +211,20 @@ async fn a_board_folded_into_dynamodb_reads_back_as_from_memory_and_no_request_i assert_eq!(got, want, "{lane:?} limit {limit:?}, seed {seed}"); } } + // The `typed` index (6j6v.xkr7): each type's live items, in id order, through the SDK. + for ty in TYPES { + let ids = |rows: Vec| -> Vec { + rows.iter() + .map(|r| text(r, "id").unwrap().to_string()) + .collect() + }; + let want = ids(mem.query_typed(ty).await.unwrap()); + let got = eventually(&want, || async move { + ids(ddb_ref.query_typed(ty).await.unwrap()) + }) + .await; + assert_eq!(got, want, "typed {ty}, seed {seed}"); + } // Two streams share the table without seeing each other: the second fold's partition // holds only its own rows (checked above), and its reads named its partition. let read = recorder.take(); @@ -257,11 +278,12 @@ async fn closing_a_ticket_with_many_dependents_costs_one_query_and_a_few_request let sent = recorder.take(); eprintln!("closing a ticket with {n} dependents sent {sent:?}"); let count = |op: &str| sent.get(op).copied().unwrap_or(0); - // The close: three changes and the watermark. The ticket's refresh: one read, one write. The - // flip: one adjacency Query, and per edge three reads — the edge, its remove, its first end, - // a dependent that is still active — and no write, since that keeps the edge on the index. + // The close: three changes and the watermark, one read to weigh the ticket's row first. The + // ticket's refresh: one read, one write. The flip: one adjacency Query, and per edge three + // reads — the edge, its remove, its first end, a dependent that is still active — and no + // write, since that keeps the edge on the index. assert_eq!(count("Query"), 1, "{sent:?}"); - assert_eq!(count("GetItem"), 1 + 3 * n, "{sent:?}"); + assert_eq!(count("GetItem"), 2 + 3 * n, "{sent:?}"); assert_eq!(count("UpdateItem"), 3 + 1 + 1, "{sent:?}"); assert_eq!(count("Scan"), 0); @@ -309,7 +331,7 @@ async fn a_snapshot_taken_from_dynamodb_starts_another_stream_with_the_same_item // The partition and the index attributes name their stream; everything else is the // same across the two. r.remove("stream_id"); - for attr in ["nxf_active", "nxf_dated"] { + for attr in ["nxf_active", "nxf_dated", "nxf_typed"] { if let Some(nxs_foundation::change::Cell::Text(v)) = r.get_mut(attr) { *v = v .replacen("stream-a", "", 1) @@ -438,3 +460,162 @@ async fn the_records_read_back_as_from_memory_with_four_requests_per_ticket_and_ ); } } + +/// The reads of a ranked list of `types` (`None`: every type) whose keys follow every edge kind, +/// read labels and custom values, the `next` board and the `list` scope of `list_types`. +fn ranked_reads(types: Option<&[&str]>, list_types: &[&str]) -> nexus_flow_facade::read::ListReads { + let mut graph = nexus_flow_facade::read::RankingReads::default(); + graph.kinds = [ + EdgeKind::Parent, + EdgeKind::ContributesTo, + EdgeKind::Mentions, + EdgeKind::Dep, + ] + .into_iter() + .collect(); + graph.labels = true; + graph.custom = true; + graph.list_types = list_types.iter().map(|t| t.to_string()).collect(); + let mut reads = nexus_flow_facade::read::ListReads::default(); + reads.types = types.map(|t| t.iter().map(|t| t.to_string()).collect()); + reads.next = true; + reads.ranking = Some(graph); + reads +} + +/// What a list board holds, comparably. +type ListView = ( + BTreeMap, + Option, + Option>, +); + +async fn list_view(t: &T, reads: &nexus_flow_facade::read::ListReads) -> ListView +where + T::Error: std::fmt::Debug, +{ + let board = records::list_board(t, reads).await.unwrap(); + (board.items, board.ranking, board.active.map(|a| a.tickets)) +} + +#[tokio::test(flavor = "multi_thread")] +async fn the_ranked_list_and_the_ranking_graph_read_back_as_from_memory() { + // Review of PR #55, Integrity #7: `records::list_board` and `records::with_ranking` through the + // SDK — the `typed` Query joined with base-table reads, `BatchGetItem` — give what they give + // over the memory table, and send no Scan. + let Some(url) = endpoint("ranked") else { + return; + }; + let recorder = Recorder::default(); + let client = client(&url, &recorder); + let name = fresh_table(&client, "ranked").await; + let folder = Folder::platform().unwrap(); + for seed in [3, 19, 41] { + let s = merged(seed); + let ops = delivered(&s, seed); + let stream = format!("stream-ranked-{seed}"); + let ddb = DynamoDbTable::new(client.clone(), &name, &stream); + let mem = MemTable::new(&stream); + folder.fold_batch(&ddb, &ops).await.unwrap(); + folder.fold_batch(&mem, &ops).await.unwrap(); + let mut all = vec![ranked_reads(None, &[])]; + for ty in TYPES { + let others: Vec<&str> = TYPES.into_iter().filter(|t| *t != ty).collect(); + all.push(ranked_reads(Some(&[ty]), &others)); + } + for reads in &all { + let want = list_view(&mem, reads).await; + let ddb_ref = &ddb; + let got = eventually(&want, || async move { list_view(ddb_ref, reads).await }).await; + assert_eq!(got, want, "seed {seed}, {:?}", reads.types); + } + let graph = ranked_reads(None, &["epic"]).ranking.unwrap(); + let plain = records::active_board(&mem, board::select(&mem).await.unwrap()) + .await + .unwrap(); + let want = records::with_ranking(&mem, plain.clone(), &graph) + .await + .unwrap() + .ranking; + let ddb_ref = &ddb; + let (plain_ref, graph_ref) = (&plain, &graph); + let got = eventually(&want, || async move { + records::with_ranking(ddb_ref, plain_ref.clone(), graph_ref) + .await + .unwrap() + .ranking + }) + .await; + assert_eq!(got, want, "seed {seed}: the ranking graph"); + let sent = recorder.take(); + assert!(sent.contains_key("BatchGetItem"), "{sent:?}"); + assert!(!sent.contains_key("Scan"), "{sent:?}"); + } +} + +#[tokio::test(flavor = "multi_thread")] +async fn a_hub_whose_adjacency_spans_query_pages_reads_every_edge() { + // Review of PR #55, Integrity #7: an item with more adjacency entries than one Query page holds + // (1 MB) — 450 contributors with ids near the key limit, some 2.9 KB an entry — reads every edge + // through the SDK, as from memory. + let Some(url) = endpoint("hub") else { return }; + let recorder = Recorder::default(); + let client = client(&url, &recorder); + let name = fresh_table(&client, "hub").await; + let folder = Folder::platform().unwrap(); + let hub = format!("ab12.h{}", "h".repeat(890)); + let mut s = Store::open_in_memory(1); + s.create_item(&hub, "epic", "Hub", "u"); + let n = 450; + for i in 0..n { + let id = format!("ab12.{i:04}{}", "n".repeat(890)); + s.create_item(&id, "task", "Contributor", "u"); + s.add_edge(&id, &hub, EdgeKind::ContributesTo, "u"); + } + let ops: Vec<(i64, _)> = s + .export() + .into_iter() + .enumerate() + .map(|(i, op)| (i as i64 + 1, op)) + .collect(); + let ddb = DynamoDbTable::new(client.clone(), &name, "stream-hub"); + let mem = MemTable::new("stream-hub"); + assert!(folder.fold_batch(&ddb, &ops).await.unwrap().is_empty()); + folder.fold_batch(&mem, &ops).await.unwrap(); + + recorder.take(); + let entries = ddb.query_prefix(&format!(".adj#{hub}#")).await.unwrap(); + let sent = recorder.take(); + assert_eq!(entries.len(), n); + assert!( + sent.get("Query").copied().unwrap_or(0) >= 2, + "the hub's entries span pages: {sent:?}" + ); + + let mut graph = nexus_flow_facade::read::RankingReads::default(); + graph.kinds.insert(EdgeKind::ContributesTo); + let plain = records::active_board(&mem, board::select(&mem).await.unwrap()) + .await + .unwrap(); + let want = records::with_ranking(&mem, plain.clone(), &graph) + .await + .unwrap() + .ranking + .unwrap(); + assert_eq!(want.edges.len(), n); + let got = records::with_ranking(&ddb, plain, &graph) + .await + .unwrap() + .ranking + .unwrap(); + assert_eq!(got, want); + + // The hub listed by its type: its edges come from the same paged entries. + let reads = ranked_reads(Some(&["epic"]), &[]); + let want = list_view(&mem, &reads).await; + let ddb_ref = &ddb; + let reads_ref = &reads; + let got = eventually(&want, || async move { list_view(ddb_ref, reads_ref).await }).await; + assert_eq!(got, want); + assert_eq!(got.1.unwrap().edges.len(), n); +} diff --git a/crates/fold-ddb/tests/fold_differential.rs b/crates/fold-ddb/tests/fold_differential.rs index ffb765f..d887b13 100644 --- a/crates/fold-ddb/tests/fold_differential.rs +++ b/crates/fold-ddb/tests/fold_differential.rs @@ -6,22 +6,23 @@ //! through the SQLite applier, in the order the replica took it in; and through this crate into a //! [`MemTable`], in a SHUFFLED order with ops delivered twice, as a relay without deduplication //! hands them over. Every row of every view table, every lane of the library and both dated lanes -//! must come out the same, and the incrementally kept indexes must be exactly what a full reindex -//! computes. +//! must come out the same, the `typed` index must list each type's live items (6j6v.xkr7), and the +//! incrementally kept indexes must be exactly what a full reindex computes. use nexus_flow_core::graph; use nexus_flow_core::model::EdgeKind; use nexus_flow_core::store::Store; use nexus_flow_core::task_reducer::TaskReducer; use nxs_fold_ddb::fold::Folder; +use nxs_fold_ddb::layout::WATERMARK_KEY; use nxs_fold_ddb::mem::{ready, MemTable}; -use nxs_fold_ddb::table::{text, Dated}; +use nxs_fold_ddb::table::{text, Dated, Table}; use nxs_fold_ddb::{board, chunks}; use nxs_foundation::reducer::Reducer; use std::collections::BTreeMap; mod common; -use common::{merged, serve, served_rows, sqlite_rows, NOW}; +use common::{merged, serve, served_rows, sqlite_rows, NOW, TYPES}; fn dated_ids(t: &MemTable, lane: Dated) -> Vec { ready(board::dated(t, lane, None)) @@ -45,6 +46,7 @@ fn the_server_fold_in_any_order_answers_as_the_local_fold() { let folder = Folder::platform().unwrap(); let mut active_seen = 0; let mut chunks_seen = 0; + let mut typed_seen = 0; for seed in 0..120 { let s = merged(seed); let t = serve(&s, seed, &folder); @@ -95,6 +97,24 @@ fn the_server_fold_in_any_order_answers_as_the_local_fold() { "archived lane, {ctx}" ); + // The `typed` index (6j6v.xkr7): under each type exactly its live items, archived ones + // included, in id order. + for ty in TYPES { + let typed: Vec = ready(t.query_typed(ty)) + .unwrap() + .iter() + .map(|r| text(r, "id").unwrap().to_string()) + .collect(); + let want = sql_ids( + &s, + &format!( + "SELECT id FROM items WHERE COALESCE(deleted,'0')<>'1' AND type='{ty}' ORDER BY id" + ), + ); + typed_seen += want.len(); + assert_eq!(typed, want, "typed {ty}, {ctx}"); + } + // The flags kept op by op are exactly what recomputing them from the rows gives. let kept = t.rows(); ready(board::reindex(&t)).unwrap(); @@ -112,6 +132,7 @@ fn the_server_fold_in_any_order_answers_as_the_local_fold() { } assert!(active_seen > 0, "the generated boards have active tickets"); assert!(chunks_seen > 0, "the generated boards supersede chunks"); + assert!(typed_seen > 0, "the generated boards have typed items"); } #[test] @@ -253,8 +274,14 @@ fn the_watermark_only_moves_forward() { ); let rows: BTreeMap<_, _> = t.rows(); assert_eq!(rows.len(), 1); - ready(Folder::clear(&t)).unwrap(); - assert!(t.rows().is_empty()); + // A clear leaves the watermark only, as a sentinel: no revision, so no reader serves the + // stream, and the position its refold has to reach. + ready(Folder::clear(&t, 10)).unwrap(); + assert_eq!(t.rows().keys().collect::>(), [WATERMARK_KEY]); + let mark = ready(Folder::watermark(&t)).unwrap().unwrap(); + assert!(mark.revisions.is_empty()); + assert_eq!((mark.folded_through, mark.refold_until), (0, Some(10))); + assert!(!folder.is_current(&mark)); } #[test] @@ -283,10 +310,12 @@ fn a_flip_costs_one_query_and_three_reads_per_edge_and_no_more() { let before = t.requests(); ready(folder.fold_batch(&t, &all(&s, seen))).unwrap(); let after = t.requests(); - // Three changes, the ticket's index write, the watermark; one read for the ticket and three - // per edge; one adjacency Query. Nothing grows with the board beyond the closed ticket's edges. + // Three changes, the ticket's index write, the watermark; two reads for the ticket — one to + // weigh its row before the op writes (review of PR #55, Integrity #4), one for its index — and + // three per edge; one adjacency Query. Nothing grows with the board beyond the closed ticket's + // edges. assert_eq!(after.writes - before.writes, 5); - assert_eq!(after.gets - before.gets, 1 + 3 * n); + assert_eq!(after.gets - before.gets, 2 + 3 * n); assert_eq!(after.queries - before.queries, 1); assert_eq!( ready(board::select(&t)) diff --git a/crates/fold-ddb/tests/list_differential.rs b/crates/fold-ddb/tests/list_differential.rs new file mode 100644 index 0000000..af8a713 --- /dev/null +++ b/crates/fold-ddb/tests/list_differential.rs @@ -0,0 +1,689 @@ +//! A server answers a ranked `list` as a replica does (6j6v.xkr7, docs/specs/derived-ranking-keys.md +//! §6 and §9 "differential"), and ranks a `next` whose keys read a `[ranking.list]` rank. +//! +//! Generated boards of four types — goals, projects, actions, persons — written by two replicas +//! apart and then merged: every status, archived and deleted items, retypes, priorities, the goals' +//! `order` custom field, `waiting` labels, and `parent`, `contributes_to`, `mentions` and `dep` +//! edges with removes. Each board is folded into a [`MemTable`] in a shuffled order with +//! duplicates, as the relay hands it over. +//! +//! The plugin ranks with every derived key kind across both sections: persons by a `sum` of +//! mentions, the `list` rank of the goals and the `next` rank of the projects they contribute to; +//! goals by their contributors under `[ranking.list]`; projects under `[ranking.next]` by the +//! `list` rank of their parent goal; actions under `[ranking.next]` by the `list` rank of the +//! persons they mention — the cross-section reads that were refused on the server until the +//! `typed` index — and, in `list`, by the `[ranking.list]` default order with a derived key. +//! +//! Three paths answer each of a set of `list` queries (types, none, container, label, status, +//! sort, limit): +//! +//! 1. the SQL path: `read::list_page` over the store; +//! 2. the Board path: `read::list_board_page` over the store's own [`read::ListBoard`] +//! (`read::list_board` for `read::list_reads`); +//! 3. the fold-ddb path: `read::list_board_page` over the folded tables' (`records::list_board`). +//! +//! All three must give the same rows, order and total, the two boards must be equal item for item +//! and edge for edge, and the JSON must be `Engine::list_query_value`'s. `next` is held to the +//! same three paths. + +use nexus_flow_core::model::{EdgeKind, ItemRow, MergeStrategy}; +use nexus_flow_core::store::Store; +use nexus_flow_facade::plugin::{self, PluginConfig, PluginRegistration}; +use nexus_flow_facade::read::{self, ListQuery, NextFilter, SortKey}; +use nxs_fold_ddb::fold::Folder; +use nxs_fold_ddb::mem::{ready, MemTable}; +use nxs_fold_ddb::{board, records}; +use std::collections::BTreeMap; + +mod common; +use common::{delivered, Lcg, NOW}; + +const ORDERS: &str = r#" +name = "list-differential-fixture" +[description] +en = "x" +de = "y" +[priority] +labels = ["P0", "P1", "P2", "P3", "P4"] +[types] +list = ["goal", "project", "action", "person"] +[vocabulary.status] +open = "open" +in_progress = "in progress" +closed = "closed" +[ranking.next] +types = ["action", "project"] +order = [ { field = "priority", dir = "asc" } ] +[ranking.next.project] +order = [ + { related = "parent", type = "goal", of = "rank", pick = "max", nulls = "first" }, + { related = "contributes_to", type = "goal", of = { field = "order" }, pick = "min", nulls = "last" }, + { count = "children", type = "action", dir = "desc" }, + { field = "priority", dir = "asc" }, +] +[ranking.next.action] +order = [ + { related = "mentions", type = "person", of = "rank", pick = "min", nulls = "last" }, + { related = "parent", type = "project", of = "rank", nulls = "last" }, + { related = "dep", of = { field = "status", precedence = ["closed", "in_progress", "open"] }, pick = "min" }, + { field = "priority", dir = "asc" }, +] +[ranking.list] +types = ["person", "goal", "project"] +order = [ + { count = "mentioned_by", dir = "desc" }, + { field = "priority", dir = "asc" }, +] +[ranking.list.goal] +order = [ + { count = "contributors", type = "project", dir = "desc" }, + { field = "priority", dir = "asc" }, +] +[ranking.list.person] +order = [ + { sum = [ { count = "mentioned_by", type = "project" }, + { count = "mentioned_by", type = "action", label = "waiting", status = ["open", "in_progress"] } ], dir = "desc" }, + { related = "contributes_to", type = "goal", of = "rank", pick = "min", nulls = "last" }, + { related = "contributes_to", type = "project", of = "rank", pick = "min", nulls = "last" }, + { count = "dependents", status = ["closed"], dir = "desc" }, + { field = "priority", dir = "asc" }, +] +[presentation.list] +columns = ["id"] +[fields.order] +type = "text" +on = ["goal"] +"#; + +inventory::submit!(PluginRegistration { + name: "list-differential-fixture", + toml: ORDERS, + order: 141, +}); + +fn cfg() -> PluginConfig { + plugin::load("list-differential-fixture").expect("the fixture is a valid plugin") +} + +const TYPES: [&str; 4] = ["goal", "project", "action", "person"]; + +/// The items of a generated board: three goals, four projects, six actions, five persons. +fn items() -> Vec<(String, &'static str)> { + let mut out = Vec::new(); + for (ty, letter, n) in [ + ("goal", 'g', 3), + ("project", 'p', 4), + ("action", 'a', 6), + ("person", 'u', 5), + ] { + for i in 0..n { + out.push((format!("ab12.{letter}{i:03}"), ty)); + } + } + out +} + +/// One replica's share of the writing. +fn write(s: &mut Store, r: &mut Lcg, added: &mut Vec<(String, String, EdgeKind)>, steps: usize) { + let all = items(); + let pick = |r: &mut Lcg| all[r.next(all.len() as u64) as usize].0.clone(); + for step in 0..steps { + s.set_wall_clock(&format!( + "2026-10-0{}T{:02}:00:00Z", + 1 + r.next(7), + step % 24 + )); + let (x, y) = (pick(r), pick(r)); + match r.next(18) { + 0 => s.set_field(&x, "status", Some("in_progress".into()), "u"), + 1 | 2 => s.set_field(&x, "status", Some("closed".into()), "u"), + 3 => s.set_field(&x, "status", Some("open".into()), "u"), + 4 => s.set_field( + &x, + "archived", + Some(format!("2026-10-0{}", 1 + r.next(7))), + "u", + ), + 5 => s.set_field(&x, "archived", None, "u"), + 6 if r.next(3) == 0 => s.delete_item(&x, "u"), + 7 => s.set_field(&x, "priority", Some(r.next(5).to_string()), "u"), + 8..=12 => { + let kind = match r.next(10) { + 0..=2 => EdgeKind::Parent, + 3..=5 => EdgeKind::ContributesTo, + 6..=8 => EdgeKind::Mentions, + _ => EdgeKind::Dep, + }; + s.add_edge(&x, &y, kind, "u"); + added.push((x, y, kind)); + } + 13 if !added.is_empty() => { + let (from, to, kind) = added.swap_remove(r.next(added.len() as u64) as usize); + s.remove_edge(&from, &to, kind, "u"); + } + 14 => { + if r.next(3) == 0 { + s.remove_label(&x, "waiting", "u"); + } else { + s.add_label(&x, "waiting", "u"); + } + } + 15 if r.next(3) == 0 => { + s.set_field(&x, "type", Some(TYPES[r.next(4) as usize].into()), "u") + } + _ => s.set_custom_field_merge( + &x, + "order", + Some(r.next(6).to_string()), + "u", + MergeStrategy::Lww, + ), + } + } +} + +/// A merged board: two replicas that wrote apart, then exchanged everything. +fn gen_board(seed: u64) -> Store { + let mut r = Lcg(seed ^ 0x115d); + let mut a = Store::open_in_memory(1); + for (id, ty) in items() { + a.create_item(&id, ty, "T", "u"); + a.set_field(&id, "priority", Some(r.next(5).to_string()), "u"); + if ty == "goal" && r.next(4) != 0 { + a.set_custom_field_merge( + &id, + "order", + Some(r.next(6).to_string()), + "u", + MergeStrategy::Lww, + ); + } + } + let mut b = Store::open_in_memory(2); + b.apply(&a.export()); + let steps = 30 + r.next(40) as usize; + write(&mut a, &mut r, &mut Vec::new(), steps); + write(&mut b, &mut r, &mut Vec::new(), steps); + let from_b = b.export(); + a.apply(&from_b); + a +} + +fn ids(items: &[ItemRow]) -> Vec { + items.iter().map(|i| i.id.clone()).collect() +} + +/// The `list` queries each board is asked: a type, several, none; a container, a label, a status; +/// the default order; a cut. +fn queries(container: &str) -> Vec { + let ranked = || ListQuery::new().sort(SortKey::Rank).now(NOW); + let types = |tys: &[&str]| tys.iter().fold(NextFilter::new(), |f, t| f.with_type(*t)); + vec![ + ranked().filter(types(&["person"])), + ranked().filter(types(&["goal"])), + ranked().filter(types(&["project", "person"])), + ranked(), + ranked().filter(types(&["action"]).with_container(container)), + ranked().filter(NextFilter::new().with_label("waiting")), + ranked().status("closed").filter(types(&["person"])), + ListQuery::new().filter(types(&["person"])), + ranked().filter(types(&["person"])).limit(3), + // Not ranked: kept and cut before the joins (review of PR #55, Integrity #1). + ListQuery::new().status("open").limit(4), + ListQuery::new() + .sort(SortKey::Closed) + .filter(NextFilter::new().with_label("waiting")) + .limit(2), + ListQuery::new() + .filter(types(&["action", "project"]).with_container(container)) + .limit(1), + ] +} + +/// What the generated boards held, so the comparison is known to have compared something worth +/// comparing. +#[derive(Default)] +struct Seen(BTreeMap<&'static str, usize>); + +impl Seen { + fn add(&mut self, what: &'static str, n: usize) { + *self.0.entry(what).or_default() += n; + } +} + +#[test] +fn the_server_answers_a_ranked_list_as_the_replica_does() { + let folder = Folder::platform().unwrap(); + let cfg = cfg(); + let mut seen = Seen::default(); + for seed in 0..40 { + let s = gen_board(seed); + let ctx = format!("seed {seed}"); + let t = MemTable::new("stream-1"); + ready(folder.fold_batch(&t, &delivered(&s, seed))).unwrap(); + + for (n, q) in queries(&format!("ab12.p{:03}", seed % 4)) + .iter() + .enumerate() + { + let ctx = format!("{ctx}, query {n}"); + let reads = read::list_reads(&cfg, q); + let local = read::list_board(&s, &reads).unwrap(); + let server = ready(records::list_board(&t, &reads)).unwrap(); + assert_eq!(server.items, local.items, "listed items, {ctx}"); + assert_eq!(server.ranking, local.ranking, "the ranking graph, {ctx}"); + assert_eq!( + server.active.as_ref().map(|a| &a.tickets), + local.active.as_ref().map(|a| &a.tickets), + "next's board, {ctx}" + ); + + // 1 = 2 = 3: the SQL path, the Board path, the fold-ddb path. + let want = read::list_page(&cfg, &s, q).unwrap(); + let board_path = read::list_board_page(&cfg, &local, q).unwrap(); + let served = read::list_board_page(&cfg, &server, q).unwrap(); + assert_eq!( + ids(&board_path.items), + ids(&want.items), + "SQL vs Board, {ctx}" + ); + assert_eq!( + ids(&served.items), + ids(&want.items), + "SQL vs fold-ddb, {ctx}" + ); + assert_eq!(served.total, want.total, "total, {ctx}"); + let records = read::list_to_value_with_custom(&cfg, &s, &want.items).unwrap(); + assert_eq!( + read::list_board_value(&cfg, &server, q).unwrap(), + read::next_page_envelope(records, &want, false), + "list JSON, {ctx}" + ); + + seen.add("listed rows", want.items.len()); + if let Some(graph) = &server.ranking { + seen.add("graph listed items", graph.listed.len()); + seen.add("graph edges", graph.edges.len()); + // What a list graph must hold and the lanes would not (review of PR #55, Test #5): + // closed and archived neighbours, inbound `contributes_to` into a listed item, and + // archived items of a graph list type, which its scope leaves out. + for n in graph.neighbours.values() { + let archived = n.item.archived.is_some(); + seen.add( + "closed neighbours in a list graph", + usize::from(!archived && n.item.status.as_deref() == Some("closed")), + ); + seen.add("archived neighbours in a list graph", usize::from(archived)); + } + seen.add( + "inbound contributes_to into a listed item", + graph + .edges + .iter() + .filter(|e| { + e.kind == EdgeKind::ContributesTo && server.items.contains_key(&e.to) + }) + .count(), + ); + let archived_in_scope: Vec = s + .list_items() + .unwrap() + .into_iter() + .filter(|i| i.archived.is_some() && i.deleted.as_deref() != Some("1")) + .filter(|i| { + i.item_type + .as_deref() + .is_some_and(|t| graph.reads.list_types.contains(t)) + }) + .map(|i| i.id) + .collect(); + assert!( + archived_in_scope + .iter() + .all(|id| !graph.listed.contains(id)), + "an archived item is out of the list scope, {ctx}" + ); + seen.add( + "archived items of a graph list type", + archived_in_scope.len(), + ); + } + seen.add( + "boards with next's board", + usize::from(server.active.is_some()), + ); + seen.add( + "boards kept and cut before their joins", + usize::from(server.cut.is_some()), + ); + seen.add( + "listed items with a closed-parent reason", + server + .items + .values() + .filter(|i| !i.parent_closed.is_empty()) + .count(), + ); + seen.add( + "closed persons listed", + want.items + .iter() + .filter(|i| { + i.item_type.as_deref() == Some("person") + && i.status.as_deref() == Some("closed") + }) + .count(), + ); + } + + // `next` whose keys read the `list` rank of persons and goals (spec §12 point 1). + let reads = read::next_ranking_reads(&cfg) + .unwrap() + .expect("derived next keys"); + let selection = ready(board::select(&t)).unwrap(); + let plain = ready(records::active_board(&t, selection)).unwrap(); + let plain = ready(records::with_contributes_to(&t, plain)).unwrap(); + let server = ready(records::with_ranking(&t, plain, &reads)).unwrap(); + let local = read::with_ranking(&s, read::active_board(&s).unwrap(), &reads).unwrap(); + assert_eq!(server.ranking, local.ranking, "next's graph, {ctx}"); + let want = read::next(&cfg, &s, NOW, None).unwrap(); + assert_eq!( + ids(&read::next_active(&cfg, &local, NOW).unwrap()), + ids(&want), + "next: SQL vs Board, {ctx}" + ); + assert_eq!( + ids(&read::next_active(&cfg, &server, NOW).unwrap()), + ids(&want), + "next: SQL vs fold-ddb, {ctx}" + ); + seen.add("next rows", want.len()); + seen.add( + "archived items", + s.list_items() + .unwrap() + .iter() + .filter(|i| i.archived.is_some() && i.deleted.as_deref() != Some("1")) + .count(), + ); + } + for what in [ + "listed rows", + "graph listed items", + "graph edges", + "boards with next's board", + "listed items with a closed-parent reason", + "closed persons listed", + "next rows", + "archived items", + "closed neighbours in a list graph", + "archived neighbours in a list graph", + "inbound contributes_to into a listed item", + "archived items of a graph list type", + "boards kept and cut before their joins", + ] { + assert!( + seen.0.get(what).copied().unwrap_or(0) > 0, + "the generated boards have {what}: {:?}", + seen.0 + ); + } +} + +#[test] +fn a_stream_not_folded_by_this_index_revision_is_refused_for_a_list() { + // The `typed` index and the self-describing adjacency entries come with index revision 3: a + // stream folded before has neither, and its list would come back empty — refused instead. + let folder = Folder::platform().unwrap(); + let cfg = cfg(); + let s = gen_board(5); + let t = MemTable::new("stream-1"); + ready(folder.fold_batch(&t, &delivered(&s, 5))).unwrap(); + use nxs_fold_ddb::fold::INDEX_DOMAIN; + use nxs_fold_ddb::layout::WATERMARK_KEY; + use nxs_fold_ddb::table::Table; + use nxs_fold_ddb::write::{Cond, Write}; + use nxs_foundation::change::Cell; + ready(t.write(&Write { + sk: WATERMARK_KEY.into(), + set: vec![(format!("revision_{INDEX_DOMAIN}"), Cell::Int(2))], + remove: Vec::new(), + condition: Some(Cond::RowPresent), + })) + .unwrap(); + let q = &queries("ab12.p000")[0]; + match ready(records::list_board(&t, &read::list_reads(&cfg, q))) { + Err(records::RecordsError::NotCurrent { found: Some(2) }) => {} + other => panic!("a revision-2 stream must be refused, got {other:?}"), + } +} + +#[test] +fn reading_a_typed_list_costs_one_index_query_three_per_item_and_one_batch() { + // §9 "bounded cost": the persons of a board are one Query on the `typed` index, and each costs + // its adjacency, labels and custom values, the timestamps of all of them one `BatchGetItem` — + // whatever else the board holds, which this list reads nothing of. A type filter reads only its + // own type. + let folder = Folder::platform().unwrap(); + let cfg = cfg(); + let mut s = Store::open_in_memory(1); + let n = 8u64; + for i in 0..n { + s.create_item(&format!("ab12.u{i:03}"), "person", "P", "u"); + } + for i in 0..30 { + s.create_item(&format!("ab12.a{i:03}"), "action", "A", "u"); + } + s.create_item("ab12.uzzz", "person", "Archived", "u"); + s.set_field("ab12.uzzz", "archived", Some("2026-10-01".into()), "u"); + let t = MemTable::new("stream-1"); + ready(folder.fold_batch(&t, &delivered(&s, 1))).unwrap(); + let q = ListQuery::new().filter(NextFilter::new().with_type("person")); + let reads = read::list_reads(&cfg, &q); + assert_eq!(reads.ranking, None); + assert!(!reads.next); + let before = t.requests(); + let server = ready(records::list_board(&t, &reads)).unwrap(); + let after = t.requests(); + assert_eq!(after.index_queries - before.index_queries, 1); + assert_eq!(after.queries - before.queries, 3 * n); + // The watermark; the persons' timestamps in one batch; no parents. + assert_eq!(after.gets - before.gets, 1); + assert_eq!(after.batch_gets - before.batch_gets, 1); + assert_eq!(after.writes, before.writes); + assert_eq!( + server.items.len(), + n as usize, + "the archived person is not listed" + ); + let page = read::list_board_page(&cfg, &server, &q).unwrap(); + assert_eq!( + ids(&page.items), + ids(&read::list_page(&cfg, &s, &q).unwrap().items) + ); +} + +#[test] +fn an_unranked_list_costs_its_page_not_the_stream() { + // Review of PR #55, Integrity #1: `list --status open --limit 5` over a stream of many closed + // items reads the joins of its five rows, not of every item. The rows' own Query is the one + // read that grows with the stream; every other request is the same with 0 closed items or 300. + let folder = Folder::platform().unwrap(); + let cfg = cfg(); + let cost = |closed: usize| { + let mut s = Store::open_in_memory(1); + for i in 0..12 { + let id = format!("ab12.a{i:03}"); + s.create_item(&id, "action", "Open", "u"); + if i % 3 == 0 { + s.add_label(&id, "waiting", "u"); + } + } + for i in 0..closed { + let id = format!("ab12.c{i:03}"); + s.create_item(&id, "action", "Closed", "u"); + s.set_field(&id, "status", Some("closed".into()), "u"); + } + let t = MemTable::new("stream-1"); + ready(folder.fold_batch(&t, &delivered(&s, 7))).unwrap(); + let mut costs = Vec::new(); + for q in [ + ListQuery::new().status("open").limit(5), + ListQuery::new() + .status("open") + .filter(NextFilter::new().with_label("waiting")) + .limit(2), + ] { + let reads = read::list_reads(&cfg, &q); + assert!(reads.select.is_some() && reads.ranking.is_none() && !reads.next); + let before = t.requests(); + let server = ready(records::list_board(&t, &reads)).unwrap(); + let after = t.requests(); + let page = read::list_board_page(&cfg, &server, &q).unwrap(); + let want = read::list_page(&cfg, &s, &q).unwrap(); + assert_eq!(ids(&page.items), ids(&want.items)); + assert_eq!(page.total, want.total, "the total counts every kept item"); + assert_eq!( + server.items.len(), + page.items.len(), + "only the page is held" + ); + costs.push(( + after.gets - before.gets, + after.batch_gets - before.batch_gets, + after.queries - before.queries, + after.index_queries - before.index_queries, + )); + } + costs + }; + let small = cost(0); + // The watermark; one `items#` Query; per page row `.adj#`, `.ladj#`, `custom_fields#`; its + // timestamps in one batch. With a label filter, each open row's labels first (12), then the + // page's adjacency and custom values. + assert_eq!(small, [(1, 1, 1 + 3 * 5, 0), (1, 1, 1 + 12 + 2 * 2, 0)]); + assert_eq!(cost(300), small, "300 closed items cost nothing more"); +} + +#[test] +fn a_ranked_list_with_next_and_a_graph_costs_what_its_scope_holds() { + // Review of PR #55, Test #7: a ranked list of persons whose keys read the `next` rank of + // projects and the `list` rank of goals reads next's board, the goals' scope and the graph. + // Its cost follows what those hold — items of no listed type, not active, on no edge, cost + // nothing. + let folder = Folder::platform().unwrap(); + let cfg = cfg(); + let q = ListQuery::new() + .sort(SortKey::Rank) + .now(NOW) + .filter(NextFilter::new().with_type("person")); + let reads = read::list_reads(&cfg, &q); + assert!(reads.next, "{reads:?}"); + let graph_reads = reads.ranking.clone().expect("a graph"); + assert!(!graph_reads.list_types.is_empty(), "{reads:?}"); + let cost = |unrelated: usize| { + let mut s = Store::open_in_memory(1); + for i in 0..4 { + s.create_item(&format!("ab12.u{i:03}"), "person", "P", "u"); + s.create_item(&format!("ab12.a{i:03}"), "action", "A", "u"); + s.add_edge( + &format!("ab12.a{i:03}"), + &format!("ab12.u{i:03}"), + EdgeKind::Mentions, + "u", + ); + } + s.create_item("ab12.g000", "goal", "G", "u"); + s.create_item("ab12.p000", "project", "P", "u"); + s.add_edge("ab12.p000", "ab12.g000", EdgeKind::ContributesTo, "u"); + s.add_edge("ab12.u000", "ab12.g000", EdgeKind::ContributesTo, "u"); + s.add_edge("ab12.u001", "ab12.p000", EdgeKind::ContributesTo, "u"); + s.add_label("ab12.a000", "waiting", "u"); + for i in 0..unrelated { + let id = format!("ab12.x{i:03}"); + s.create_item(&id, "action", "Done", "u"); + s.set_field(&id, "status", Some("closed".into()), "u"); + } + let t = MemTable::new("stream-1"); + ready(folder.fold_batch(&t, &delivered(&s, 3))).unwrap(); + let before = t.requests(); + let server = ready(records::list_board(&t, &reads)).unwrap(); + let after = t.requests(); + let page = read::list_board_page(&cfg, &server, &q).unwrap(); + assert_eq!( + ids(&page.items), + ids(&read::list_page(&cfg, &s, &q).unwrap().items) + ); + let graph = server.ranking.as_ref().unwrap(); + assert!(!graph.edges.is_empty() && !graph.listed.is_empty()); + let spent = ( + after.gets - before.gets, + after.batch_gets - before.batch_gets, + after.queries - before.queries, + after.index_queries - before.index_queries, + ); + // Every point read is the watermark or a batch; one index Query per listed type, the + // `active` index and one per graph list type. + assert_eq!(spent.0, 1); + assert!(spent.1 <= 3, "{spent:?}"); + assert_eq!(spent.3, 1 + 1 + graph_reads.list_types.len() as u64); + spent + }; + let small = cost(0); + assert_eq!( + cost(200), + small, + "200 unrelated closed actions cost nothing more" + ); +} + +#[test] +fn an_item_whose_typed_attribute_lags_its_row_is_not_listed_under_its_old_type() { + // Review of PR #55, Code #2 / Test #3: the fold writes `nxf_typed` after the row, so a run that + // died between the two leaves a row retyped or deleted whose attribute still names its old + // type. The list asks the index's copy of the row, not the attribute alone. + use nxs_fold_ddb::table::{text, Table}; + use nxs_fold_ddb::write::{Cond, Write}; + use nxs_foundation::change::Cell; + let folder = Folder::platform().unwrap(); + let cfg = cfg(); + let mut s = Store::open_in_memory(1); + for i in 0..3 { + s.create_item(&format!("ab12.u{i:03}"), "person", "P", "u"); + } + let t = MemTable::new("stream-1"); + ready(folder.fold_batch(&t, &delivered(&s, 1))).unwrap(); + for (id, column, value) in [("ab12.u001", "type", "goal"), ("ab12.u002", "deleted", "1")] { + ready(t.write(&Write { + sk: format!("items#{id}"), + set: vec![(column.into(), Cell::Text(value.into()))], + remove: Vec::new(), + condition: Some(Cond::RowPresent), + })) + .unwrap(); + } + let on_index = |ty: &str| -> Vec { + ready(t.query_typed(ty)) + .unwrap() + .iter() + .map(|r| text(r, "id").unwrap().to_string()) + .collect() + }; + assert_eq!( + on_index("person").len(), + 3, + "the attributes still name `person`" + ); + let listed = |ty: &str| -> Vec { + let q = ListQuery::new() + .sort(SortKey::Rank) + .filter(NextFilter::new().with_type(ty)); + let board = ready(records::list_board(&t, &read::list_reads(&cfg, &q))).unwrap(); + board.items.into_keys().collect() + }; + assert_eq!(listed("person"), ["ab12.u000"]); + assert!( + listed("goal").is_empty(), + "nor under its new type, until the attribute moves" + ); +} diff --git a/crates/fold-ddb/tests/records_differential.rs b/crates/fold-ddb/tests/records_differential.rs index 99ca095..b300b93 100644 --- a/crates/fold-ddb/tests/records_differential.rs +++ b/crates/fold-ddb/tests/records_differential.rs @@ -22,7 +22,7 @@ use nxs_fold_ddb::{board, records}; use nxs_foundation::change::Cell; mod common; -use common::{delivered, id, merged, serve, Lcg, NOW}; +use common::{delivered, head, id, merged, serve, Lcg, NOW}; /// A plugin that declares the generator's custom field, so the `custom` join is exercised, and /// ranks by priority and due date. @@ -440,7 +440,7 @@ fn a_stream_folded_before_the_entries_is_refused_then_refolded_and_reads_its_lab // stream and fold it again from position 0. No step of its own. let mark = ready(Folder::watermark(&t)).unwrap().unwrap(); assert!(!folder.is_current(&mark), "seed {seed}"); - ready(Folder::clear(&t)).unwrap(); + ready(Folder::clear(&t, head(&ops))).unwrap(); ready(folder.fold_batch(&t, &ops)).unwrap(); let mark = ready(Folder::watermark(&t)).unwrap().unwrap(); assert!(folder.is_current(&mark), "seed {seed}"); @@ -634,7 +634,7 @@ fn a_stream_folded_by_index_revision_1_is_refused_then_refolded_and_reads_its_co } let mark = ready(Folder::watermark(&t)).unwrap().unwrap(); assert!(!folder.is_current(&mark), "seed {seed}"); - ready(Folder::clear(&t)).unwrap(); + ready(Folder::clear(&t, head(&ops))).unwrap(); ready(folder.fold_batch(&t, &ops)).unwrap(); let served = served_board(&t); diff --git a/crates/fold-ddb/tests/robustness.rs b/crates/fold-ddb/tests/robustness.rs index 414a3f3..46279a3 100644 --- a/crates/fold-ddb/tests/robustness.rs +++ b/crates/fold-ddb/tests/robustness.rs @@ -1,21 +1,28 @@ //! What a server fold does when things go wrong (review of PR #24): a run that dies at any write //! and folds its batch again, an op the table cannot hold, a clear that dies half way, a release -//! with other fold revisions, a lane cut by a limit, and a snapshot that does not fit. +//! with other fold revisions, a lane cut by a limit, a snapshot that does not fit — and the `typed` +//! index kept through retypes, archives and deletes (6j6v.xkr7); a stream being refolded, a row +//! with no room for its index attributes and the escaping of a `typed` partition (review of PR +//! #55). mod common; -use common::{delivered, merged, NOW}; +use common::{delivered, head, merged, NOW}; use nexus_flow_core::model::EdgeKind; use nexus_flow_core::store::Store; +use nexus_flow_facade::read; use nxs_fold_ddb::board; use nxs_fold_ddb::fold::{Folded, Folder}; -use nxs_fold_ddb::layout::WATERMARK_KEY; +use nxs_fold_ddb::layout::{typed_partition, MAX_ITEM_BYTES, MAX_PARTITION_BYTES, WATERMARK_KEY}; use nxs_fold_ddb::mem::{ready, MemTable}; +use nxs_fold_ddb::records::{self, RecordsError}; use nxs_fold_ddb::snapshot::{self, ImportError}; use nxs_fold_ddb::table::{text, Dated, Outcome, Row, Table}; +use nxs_fold_ddb::write::attribute_bytes; use nxs_fold_ddb::write::{Cond, Write}; use nxs_foundation::change::Cell; use nxs_foundation::model::Op; +use std::collections::BTreeMap; use std::sync::atomic::{AtomicUsize, Ordering}; #[derive(Debug)] @@ -72,6 +79,10 @@ impl Table for Failing<'_> { Ok(ok(self.inner.get(sk).await)) } + async fn get_many(&self, sks: &[String]) -> Result, Injected> { + Ok(ok(self.inner.get_many(sks).await)) + } + async fn write(&self, write: &Write) -> Result { self.tick()?; Ok(ok(self.inner.write(write).await)) @@ -94,6 +105,10 @@ impl Table for Failing<'_> { async fn query_dated(&self, lane: Dated, limit: Option) -> Result, Injected> { Ok(ok(self.inner.query_dated(lane, limit).await)) } + + async fn query_typed(&self, item_type: &str) -> Result, Injected> { + Ok(ok(self.inner.query_typed(item_type).await)) + } } /// Fold `first` cleanly, then `second` with a run that dies at each write in turn and folds @@ -191,7 +206,8 @@ fn an_op_the_table_cannot_hold_is_refused_and_the_stream_folds_on() { let refused = ready(folder.fold_batch(&t, &ops)).unwrap(); let reasons: Vec<&str> = refused.iter().map(|r| r.reason.as_str()).collect(); // The long id's own three field ops and its edge (whose adjacency key is too long); the 500 KB - // description; and the design that would grow ab12.0002 past the limit. + // description; and the design that would grow ab12.0002 past the limit — weighed with the row + // it lands on before anything is written (review of PR #55, Integrity #4). assert_eq!(refused.len(), 6, "{reasons:#?}"); assert_eq!( reasons.iter().filter(|r| r.contains("a key of")).count(), @@ -199,7 +215,12 @@ fn an_op_the_table_cannot_hold_is_refused_and_the_stream_folds_on() { "{reasons:#?}" ); assert!(reasons.iter().any(|r| r.contains("a row of"))); - assert!(reasons.iter().any(|r| r.contains("kept their state"))); + assert!( + reasons + .iter() + .any(|r| r.contains("item ab12.0002 would weigh")), + "{reasons:#?}" + ); // The stream folded on: the watermark moved past every op, and what fits is there. let mark = ready(Folder::watermark(&t)).unwrap().unwrap(); assert_eq!(mark.folded_through, ops.len() as i64); @@ -214,6 +235,130 @@ fn an_op_the_table_cannot_hold_is_refused_and_the_stream_folds_on() { assert_eq!(lanes.ready, vec!["ab12.0001", "ab12.0002"]); } +#[test] +fn a_type_too_long_for_the_typed_index_is_refused_and_the_item_folds_on_without_it() { + // 6j6v.xkr7: the `typed` index partition carries the type; DynamoDB refuses an index key past + // 2,048 bytes, and a refused write would fail every run at the same op. So the op that sets + // such a type is refused before it writes, as an op with a key too long is, and the item keeps + // the type it had. + let folder = Folder::platform().unwrap(); + let mut s = Store::open_in_memory(1); + s.create_item("ab12.0001", "task", "T", "u"); + s.set_field("ab12.0001", "type", Some("t".repeat(2100)), "u"); + s.create_item("ab12.0002", &"n".repeat(2030), "Just fits", "u"); + let t = MemTable::new("stream-1"); + let refused = ready(folder.fold_batch(&t, &numbered(&s, 0))).unwrap(); + let reasons: Vec<&str> = refused.iter().map(|r| r.reason.as_str()).collect(); + assert_eq!(refused.len(), 1, "{reasons:#?}"); + assert!( + reasons[0].contains("`typed` index partition"), + "{reasons:#?}" + ); + let ids = |ty: &str| -> Vec { + ready(t.query_typed(ty)) + .unwrap() + .iter() + .map(|r| text(r, "id").unwrap().to_string()) + .collect() + }; + assert_eq!(ids("task"), ["ab12.0001"]); + assert_eq!(ids(&"n".repeat(2030)), ["ab12.0002"]); +} + +#[test] +fn retyping_archiving_and_deleting_move_an_item_on_the_typed_index() { + // 6j6v.xkr7: an item is listed under its type while it is live — archived or not — moves with a + // retype, and leaves when it is deleted. A type with `#` in it gets its own partition. + let folder = Folder::platform().unwrap(); + let mut s = Store::open_in_memory(1); + s.create_item("ab12.0001", "person", "Ada", "u"); + s.create_item("ab12.0002", "person", "Bob", "u"); + s.create_item("ab12.0003", "a#b", "Odd", "u"); + let t = MemTable::new("stream-1"); + let mut seen = 0; + let mut step = |s: &Store| { + let ops = numbered(s, seen); + seen += ops.len(); + ready(folder.fold_batch(&t, &ops)).unwrap(); + ["person", "project", "a#b", "a"].map(|ty| { + ready(t.query_typed(ty)) + .unwrap() + .iter() + .map(|r| text(r, "id").unwrap().to_string()) + .collect::>() + }) + }; + let v = |ids: &[&str]| ids.iter().map(|s| s.to_string()).collect::>(); + let none = v(&[]); + + assert_eq!( + step(&s), + [ + v(&["ab12.0001", "ab12.0002"]), + none.clone(), + v(&["ab12.0003"]), + none.clone() + ] + ); + s.set_field("ab12.0002", "archived", Some("2026-10-08".into()), "u"); + s.set_field("ab12.0001", "status", Some("closed".into()), "u"); + assert_eq!( + step(&s), + [ + v(&["ab12.0001", "ab12.0002"]), + none.clone(), + v(&["ab12.0003"]), + none.clone() + ], + "closed and archived items stay listed" + ); + s.set_field("ab12.0001", "type", Some("project".into()), "u"); + assert_eq!( + step(&s), + [ + v(&["ab12.0002"]), + v(&["ab12.0001"]), + v(&["ab12.0003"]), + none.clone() + ] + ); + s.delete_item("ab12.0002", "u"); + s.delete_item("ab12.0003", "u"); + assert_eq!( + step(&s), + [none.clone(), v(&["ab12.0001"]), none.clone(), none.clone()] + ); + // Created again under a deleted id, and a type cleared to none: the index lists what the store + // holds live, by type — whatever the store makes of the re-create — and an item without a type + // is on no partition (review of PR #55, Test #6). + s.create_item("ab12.0002", "person", "Bob again", "u"); + s.set_field("ab12.0001", "type", None, "u"); + let live = |s: &Store| { + ["person", "project", "a#b", "a"].map(|ty| { + let mut ids: Vec = s + .list_items() + .unwrap() + .into_iter() + .filter(|i| i.deleted.as_deref() != Some("1")) + .filter(|i| i.item_type.as_deref() == Some(ty)) + .map(|i| i.id) + .collect(); + ids.sort(); + ids + }) + }; + let want = live(&s); + assert_eq!(want[1], none, "the type cleared to none leaves `project`"); + assert_eq!(step(&s), want); + assert!(!t.rows()["items#ab12.0001"].contains_key("nxf_typed")); + // The attribute is on the item rows only, never on an adjacency entry or another row. + assert!(t + .rows() + .iter() + .filter(|(_, r)| r.contains_key("nxf_typed")) + .all(|(sk, _)| sk.starts_with("items#"))); +} + #[test] fn a_clear_that_dies_keeps_the_watermark_and_a_release_with_other_rules_refolds_from_zero() { let folder = Folder::platform().unwrap(); @@ -235,16 +380,27 @@ fn a_clear_that_dies_keeps_the_watermark_and_a_release_with_other_rules_refolds_ let mark = ready(Folder::watermark(&t)).unwrap().unwrap(); assert!(!folder.is_current(&mark)); - // A clear that dies half way leaves the watermark — the next run sees the old revisions again. - assert!(ready(Folder::clear(&Failing::new(&t, 3))).is_err()); + // A clear that dies half way leaves a watermark that is not current — the next run clears it + // again. + assert!(ready(Folder::clear(&Failing::new(&t, 3), head(&ops))).is_err()); let after = ready(Folder::watermark(&t)).unwrap().unwrap(); assert!(!folder.is_current(&after)); - // Cleared for good and folded from position 0, it holds what a fresh fold holds. - ready(Folder::clear(&t)).unwrap(); - assert!(t.rows().is_empty()); + // Cleared for good, only the sentinel watermark is left; folded from position 0, the stream + // holds what a fresh fold holds, its watermark current and caught up with the clear's head. + ready(Folder::clear(&t, head(&ops))).unwrap(); + assert_eq!(t.rows().keys().collect::>(), [WATERMARK_KEY]); ready(folder.fold_batch(&t, &ops)).unwrap(); - assert_eq!(t.rows(), clean.rows()); + let without_mark = |t: &MemTable| { + let mut rows = t.rows(); + rows.remove(WATERMARK_KEY); + rows + }; + assert_eq!(without_mark(&t), without_mark(&clean)); + let mark = ready(Folder::watermark(&t)).unwrap().unwrap(); + assert!(folder.is_current(&mark)); + assert_eq!(mark.folded_through, head(&ops)); + assert_eq!(mark.refold_until, Some(head(&ops))); } #[test] @@ -382,11 +538,17 @@ fn a_snapshot_that_does_not_fit_is_refused_before_anything_is_written() { column.tables.get_mut("items").unwrap()[0].insert("nope".into(), serde_json::json!("x")); assert!(matches!(refused(&column), ImportError::Layout(_))); - // An import that died after some rows — no watermark yet — is simply run again. + // An import that died after some rows is refused to readers — its watermark is the sentinel + // it wrote first, naming no revision — and is simply run again. let t = MemTable::new("stream-1"); let partial = ready(snapshot::import(&Failing::new(&t, 10), &folder, &good)); assert!(partial.is_err()); - assert!(ready(Folder::watermark(&t)).unwrap().is_none()); + let mark = ready(Folder::watermark(&t)).unwrap().unwrap(); + assert!(mark.revisions.is_empty() && !folder.is_current(&mark)); + assert!(matches!( + ready(records::list_board(&t, &read::ListReads::default())), + Err(RecordsError::NotCurrent { found: None }) + )); ready(snapshot::import(&t, &folder, &good)).unwrap(); let whole = MemTable::new("stream-1"); ready(snapshot::import(&whole, &folder, &good)).unwrap(); @@ -396,3 +558,176 @@ fn a_snapshot_that_does_not_fit_is_refused_before_anything_is_written() { Folded::Folded )); } + +/// What the three readers of a hosted board answer for `t`: the records, the list, the ranking +/// graph (`plain` read before). +fn served( + t: &MemTable, + plain: &read::ActiveBoard, +) -> [Result<(), RecordsError>; 3] { + let selection = ready(board::select(t)).unwrap(); + [ + ready(records::active_board(t, selection)).map(drop), + ready(records::list_board(t, &read::ListReads::default())).map(drop), + ready(records::with_ranking( + t, + plain.clone(), + &read::RankingReads::default(), + )) + .map(drop), + ] +} + +#[test] +fn a_refolding_stream_is_refused_from_its_clear_until_the_refold_reaches_the_head() { + // Review of PR #55, Integrity #2: between a clear and the end of its refold the tables hold part + // of the history. Every reader refuses the stream for that whole window — `NotCurrent` while + // the watermark names no revision, `Refolding` once the first batch wrote them — and serves it + // again exactly when the refold reaches the head the clear named. + let folder = Folder::platform().unwrap(); + let s = merged(4); + let ops = delivered(&s, 4); + let t = MemTable::new("stream-1"); + ready(folder.fold_batch(&t, &ops)).unwrap(); + let plain = ready(records::active_board(&t, ready(board::select(&t)).unwrap())).unwrap(); + assert!(!plain.tickets.is_empty()); + assert!(served(&t, &plain).iter().all(Result::is_ok)); + let whole = t.rows(); + + // A clear that dies after its sentinel: refused, and not current, so the next run clears again. + assert!(ready(Folder::clear(&Failing::new(&t, 1), head(&ops))).is_err()); + for answer in served(&t, &plain) { + assert!( + matches!(answer, Err(RecordsError::NotCurrent { found: None })), + "{answer:?}" + ); + } + assert!(!folder.is_current(&ready(Folder::watermark(&t)).unwrap().unwrap())); + + ready(Folder::clear(&t, head(&ops))).unwrap(); + for answer in served(&t, &plain) { + assert!( + matches!(answer, Err(RecordsError::NotCurrent { found: None })), + "{answer:?}" + ); + } + + // Half the history refolded: this build's revisions are written, the head is not reached. + let half = ops.len() / 2; + ready(folder.fold_batch(&t, &ops[..half])).unwrap(); + assert!(folder.is_current(&ready(Folder::watermark(&t)).unwrap().unwrap())); + for answer in served(&t, &plain) { + match answer { + Err(RecordsError::Refolding { + folded_through, + until, + }) => assert_eq!((folded_through, until), (head(&ops[..half]), head(&ops))), + other => panic!("a half-refolded stream must be refused, got {other:?}"), + } + } + + // Caught up: served again, the same tables as before the clear; a batch folded twice changes + // nothing. + ready(folder.fold_batch(&t, &ops[half..])).unwrap(); + assert!(served(&t, &plain).iter().all(Result::is_ok)); + ready(folder.fold_batch(&t, &ops[half..])).unwrap(); + assert!(served(&t, &plain).iter().all(Result::is_ok)); + let without_mark = |mut rows: BTreeMap| { + rows.remove(WATERMARK_KEY); + rows + }; + assert_eq!(without_mark(t.rows()), without_mark(whole)); +} + +#[test] +fn a_typed_partition_escapes_the_type_so_no_stream_and_type_share_one() { + // Review of PR #55, Test #2: the stream is the partition's prefix and the type its rest, so a + // `#` in the type must not read as the separator. + assert_ne!(typed_partition("a", "b#c"), typed_partition("a#b", "c")); + assert_eq!(typed_partition("a", "b#c"), "a#b%23c"); + assert_ne!(typed_partition("a", "b%23c"), typed_partition("a", "b#c")); + + // A type of `#`s triples when escaped: it is weighed escaped, so a type whose own bytes fit is + // refused once its partition would not. + let folder = Folder::platform().unwrap(); + let hashes = "#".repeat(700); + assert!("stream-1#".len() + hashes.len() <= MAX_PARTITION_BYTES); + assert!(typed_partition("stream-1", &hashes).len() > MAX_PARTITION_BYTES); + let mut s = Store::open_in_memory(1); + s.create_item("ab12.0001", &hashes, "Hashes", "u"); + s.create_item("ab12.0002", &"x".repeat(700), "Plain", "u"); + let t = MemTable::new("stream-1"); + let refused = ready(folder.fold_batch(&t, &numbered(&s, 0))).unwrap(); + let reasons: Vec<&str> = refused.iter().map(|r| r.reason.as_str()).collect(); + assert_eq!(refused.len(), 1, "{reasons:#?}"); + assert!( + reasons[0].contains("`typed` index partition"), + "{reasons:#?}" + ); + assert!(ready(t.query_typed(&hashes)).unwrap().is_empty()); + let plain: Vec = ready(t.query_typed(&"x".repeat(700))) + .unwrap() + .iter() + .map(|r| text(r, "id").unwrap().to_string()) + .collect(); + assert_eq!(plain, ["ab12.0002"]); +} + +#[test] +fn an_item_with_no_room_for_its_index_attributes_is_refused_rather_than_left_off_an_index() { + // Review of PR #55, Integrity #4: closing a ticket swaps its `active` attribute for the two + // `dated` ones, which weigh more. A row the close itself fits, but not with those, used to take + // the close and drop the attribute write in silence — closed in its cells, still on `active`, + // on no `dated` lane. The op is refused before it writes anything instead. + let folder = Folder::platform().unwrap(); + let stream = "stream-1"; + let ops_with = |description: usize| { + let mut s = Store::open_in_memory(1); + s.create_item("ab12.0001", "task", "T", "u"); + s.set_field( + "ab12.0001", + "description", + Some("d".repeat(description)), + "u", + ); + s.set_field("ab12.0001", "status", Some("closed".into()), "u"); + numbered(&s, 0) + }; + let weigh = |r: &Row| r.iter().map(|(n, c)| attribute_bytes(n, c)).sum::(); + + // Measured on a small row: the closed row with its index attributes (`typed`, `dated`), and as + // it would weigh still carrying the open row's (`typed`, `active`). + let small = MemTable::new(stream); + ready(folder.fold_batch(&small, &ops_with(10))).unwrap(); + let row = small.rows()["items#ab12.0001"].clone(); + let dated: usize = ["nxf_dated", "nxf_dated_at"] + .iter() + .map(|a| attribute_bytes(a, &row[*a])) + .sum(); + let active = attribute_bytes("nxf_active", &Cell::Text(stream.into())); + assert!(dated > active, "the close makes the row heavier"); + let with_open_attributes = weigh(&row) - dated + active; + + // The description that leaves the closed row exactly at the limit with the open row's + // attributes: the close itself fits, its index attributes do not. + let ops = ops_with(10 + MAX_ITEM_BYTES - with_open_attributes); + let t = MemTable::new(stream); + let (close, before) = ops.split_last().unwrap(); + assert!(ready(folder.fold_batch(&t, before)).unwrap().is_empty()); + match ready(folder.fold(&t, &close.1)).unwrap() { + Folded::Refused(r) => assert!(r.reason.contains("index attributes"), "{r:?}"), + other => panic!("the close must be refused, got {other:?}"), + } + // Refused whole: still open in its cells and on its indexes, on no `dated` lane. + let row = &t.rows()["items#ab12.0001"]; + assert_eq!(text(row, "status"), Some("open")); + assert!(row.contains_key("nxf_active") && row.contains_key("nxf_typed")); + assert!(weigh(row) <= MAX_ITEM_BYTES); + assert!(ready(board::dated(&t, Dated::Closed, None)) + .unwrap() + .is_empty()); + assert_eq!( + ready(board::select(&t)).unwrap().board.lanes(NOW).ready, + vec!["ab12.0001"] + ); +} diff --git a/docs/specs/E4-sync-protocol-server.md b/docs/specs/E4-sync-protocol-server.md index a63c679..01d63b0 100644 --- a/docs/specs/E4-sync-protocol-server.md +++ b/docs/specs/E4-sync-protocol-server.md @@ -278,6 +278,13 @@ relay — does it with `nxs-fold-ddb`: the reducers a local replica registers, t - **GSI `dated`** (`nxf_dated` / `nxf_dated_at`): sparse — `#closed` by `closed_at`, `#archived` by `archived`, newest first; the sort key is `U+001F`, so tickets of one instant keep one order and a `limit` cuts the lane at the same place every time. +- **GSI `typed`** (`nxf_typed` / `sk`, both S, projecting all; added 2026-10-11, `6j6v.xkr7`): + sparse — `#` (the type escaped as a key value: `%` → `%25`, `#` → `%23`) on every + live item row, closed and archived ones included, removed on a delete. One Query lists a type's + live items, the `list` scope a ranked hosted `list` and the derived ranking keys rank over + (`records::list_board`, `docs/specs/derived-ranking-keys.md` §6). The deploying stack adds the + GSI first: a fold against a table without it still writes the attribute harmlessly, and only + the read needs the index. - **The active tickets' records** (added 2026-10-10, `6j6v.t1ym`). The LANES need only the `active` Query above. The RECORDS a lane is shown with need more: a lane's JSON carries joins kept in other tables — labels, custom values, timestamps, bearing thread links, the parent's @@ -297,14 +304,43 @@ relay — does it with `nxs-fold-ddb`: the reducers a local replica registers, t reducers' rows — flags, adjacency entries, entries under a ticket — is versioned in the watermark as `revision_index` (1: the entries under a ticket). A stream folded before has no such revision, so `Folder::is_current` is false and the fold run clears and refolds it from position 0, exactly - as for a release with other reducer revisions — no manual step. Until then `active_board` - refuses the stream (`RecordsError::NotCurrent`) rather than answer without its labels. + as for a release with other reducer revisions — no manual step. Until then every records read + (`active_board`, `with_ranking`, `list_board`) refuses the stream (`RecordsError::NotCurrent`) + rather than answer without its labels — and goes on refusing it through the refold (review of PR + #55): `Folder::clear(table, until)` rewrites the watermark FIRST as a sentinel that names no + revision, `folded_through` 0 and `refold_until` = `until`, the relay head the refold must reach; + the refold's first batch writes the revisions, and the reads answer `RecordsError::Refolding` + until `folded_through` reaches `until`. Nothing has to be written for that, so a refold that + dies and resumes, or folds a batch twice, ends served the same. A snapshot import writes the same + sentinel before its first row. **No mixed fold versions:** `is_current` is equality, so a build + with older rules also clears a stream folded by newer ones — roll a revision bump out to every + fold run at once, and roll back the same way. +- **What a hosted read costs, per call** (review of PR #55, Integrity #1). One `GetItem` of the + watermark per read. `active_board`: `4n + p` more for `n` active tickets and `p` parents + outside them. `with_ranking`: one Query per active ticket, the neighbours' rows by `BatchGetItem` + (100 keys a request, unprocessed keys sent again), and per live neighbour up to three Queries; + a `next` key that reads a `[ranking.list]` rank adds one `typed` Query per such type and up to + three Queries per live, unarchived item of it. `list_board`: one `typed` Query per listed type + (or one `items#` Query, paged at 1 MB, without a type filter); a list that is NOT ranked keeps + and cuts on the rows first (`ListReads::select`), so beyond the rows it costs three Queries per + page row, one per kept row for a container or label filter, and one `BatchGetItem` per 100 + timestamps and outside parents — `list --limit 10` about 33 requests, whatever the stream + holds. A RANKED list reads three Queries per live, unarchived item of its types plus the batched + point reads, and `next`'s board and the graph when its keys read them: a rank is computed over + the type's whole `list` scope (`docs/specs/derived-ranking-keys.md` §12), so that cost is the + scope's, not the answer's. No read caps its requests; a hard per-call budget that answers with an + explicit error is filed as `6j6v.wq9e`. At most 16 items are read at a time + (`READ_CONCURRENCY`). - **Index reads are eventually consistent**; row reads are strong. The lanes right after a fold may be those of the state before it. - **An op the table cannot hold** (a sort key past 1,024 bytes, a row past 400 KB) is refused and reported, not retried for ever; the relay keeps it. A row that only outgrows the limit through several ops keeps its state at the write that would break it — the one result that depends on - arrival order, and only for rows a local replica can hold but DynamoDB cannot. + arrival order, and only for rows a local replica can hold but DynamoDB cannot. An item row is + weighed BEFORE the op writes anything — stored row, the op's writes and the index attributes it + would then carry (`nxf_active`, `nxf_typed`, `nxf_dated`/`nxf_dated_at`), one `GetItem` — so an + item never takes a change its index attributes do not fit beside and drops off an index in + silence (review of PR #55, Integrity #4). - **Edge flags are rewritten before the ticket's own flag**, so a run that dies between the two still sees the flip when it folds the op again. - **No Scan.** Every read names its partition. The `dynamodb-local` CI job records every request a @@ -316,7 +352,9 @@ relay — does it with `nxs-fold-ddb`: the reducers a local replica registers, t **The watermark** (`.meta#watermark`) is `folded_through` — a relay position, per decision 4 above — plus each reducer's fold revision. A batch folds, then moves the mark forward; a run that dies in between folds part of a batch again, which changes nothing. A release whose revisions differ clears -the stream — the watermark last, so a clear that dies is done again — and folds it from position 0. +the stream — the watermark first, rewritten as a sentinel without revisions, so a clear that dies +is done again and no reader serves the stream until its refold reaches the head the clear named — +and folds it from position 0. **A snapshot is tables plus watermark, without the log** — only to start a fold run. It is taken without a lock, the watermark first and the tables after, so the tables may hold ops past the mark diff --git a/docs/specs/derived-ranking-keys.md b/docs/specs/derived-ranking-keys.md index 22c2d7f..536cdc2 100644 --- a/docs/specs/derived-ranking-keys.md +++ b/docs/specs/derived-ranking-keys.md @@ -2,7 +2,10 @@ **Status: decided by the owner on 2026-10-09** (§10). The cuts that build it are listed in §11. **Built locally (x0wf, §12):** the three key kinds, their load-time rules and the type chain over -the SQL path, in `next` and `list`. The server cuts (1c5s, xkr7) are open. +the SQL path, in `next` and `list`. **Built on the server for `next` (1c5s, §6):** the same keys +over the hosted board, with the rule the owner decided on 2026-10-11 (§10.5). **Built on the server +for `list` (xkr7, §6):** the `typed` index, a ranked hosted `list`, and a hosted `next` whose keys +read a `[ranking.list]` rank. Fourth cut of `6j6v.kfs7` (cut with the owner on 2026-10-09). The three cuts before it add the plumbing this one ranks with: @@ -135,16 +138,55 @@ derived key pays nothing. `nxs-fold-ddb` answers `next` from a sparse index of ACTIVE tickets through `graph::Board` (6j6v.k7w7). Derived keys must give the same order there. -**`next`.** -- `Board` gains the edge kinds the keys need (`contributes_to`, `mentions`), and each ticket's - type, status, labels and the fields the keys read. -- The rule that makes this possible is the one `Board` already has: a counterpart that is not an - active ticket counts as absent, which is exactly §4's scope for `next`. - -**`list` needs a second index (decided §10.4).** Objects that `list` ranks, such as a person, are +**`next`** (built in 1c5s; decided by the owner on 2026-10-11, §10.5). +- **A key reads every LIVE neighbour, on the server as locally** (§12, "Live neighbours"): a + deleted neighbour is gone, a closed or archived one counts. This replaces the first reading of + this section, that a counterpart which is not an active ticket counts as absent. That rule stays + the rule of the LANES (`graph::Board`); it is not the rule of the keys. Only the scope of + `of = "rank"` is the active set: a neighbour outside `next`'s candidates has no rank (§4), on + both paths. +- **Where it lives.** The facade's `ActiveBoard` — the store-free lanes' input, `graph::Board` + plus each active ticket's row and joins — gains a `RankingGraph`: every present edge of the kinds + the keys name (`contributes_to`, `mentions`, `dep`, `parent` as the `present_parent` projection) + with an active end and both ends live, and each live neighbour that is not active with its row, + labels and custom values. `graph::Board` itself is unchanged: it holds no rows, and the lanes read + no neighbour that is not active. `read::next_ranking_reads(cfg)` names what the graph must hold + for a plugin; `read::next_active` refuses a plugin with derived keys while its board lacks it, and + ranks by the same evaluation the store path runs once it is there. +- **What the fold writes.** Every edge add already had an adjacency entry under each end + (`.adj##`), whatever its kind and its ends' activity. Since index revision 3 each entry + also copies the add's `from_id`, `to_id`, `kind`, `lamport` and `site` and says whether the add is + removed, kept up to date when the remove is folded. So one Query on an item's prefix reads every + edge touching it, in and out, with its presence: inbound `contributes_to`, `mentions` both ways, + and the `parent` edges a child's current parent is picked from. No new index, no new key + attribute: 1c5s alone leaves the table contract unchanged — the release that ships it changes it + all the same, with xkr7's GSI `typed` below. Revision 3 makes every stream folded before it not + current; it answers `NotCurrent` from the moment the fold run clears it, and `Refolding` until + the refold reaches the relay position the clear named (review of PR #55: `Folder::clear` leaves + a sentinel watermark without revisions and that position, so a half-refolded stream is never + served). +- **What the server reads** (`records::with_ranking`): one `GetItem` of the watermark (the read + refuses a stream that is not current or still refolding itself); one Query per active ticket on + its adjacency; the rows of the neighbours that are not active by `BatchGetItem` (100 keys a + request) and, per live one, one Query on its own adjacency for its current parent, plus one Query + each for its labels and custom values when a key reads them. +- **A `[ranking.list]` rank in a `next` key** (refused on the server in 1c5s, ranked since xkr7): a + `[ranking.next]` order that reads — directly or down its type chain — the rank of a type only + `[ranking.list]` orders (§12, first point). That rank is computed over the type's live, + unarchived items, which the active board does not hold; `read::next_ranking_reads` names such a + type in `RankingReads::list_types`, and the server reads its items off the `typed` index into the + graph (`RankingGraph::listed`), with the edges around them. +- **What xkr7 builds on.** The self-describing adjacency entries are written for EVERY edge between + ANY items, so xkr7's "widened adjacency" needed no new entries for the edges: a `list`-ranked item + reads its edges with the same Query (verified in xkr7: its `.adj#` entries give its current + parent, its `contributes_to` targets, its parents for `parent_closed_reason` and every edge a key + follows or counts). What xkr7 adds is the `typed` index (the `list` scope of a type), the read + that ranks a type's live items from it, and lifting the refusal above. + +**`list` needs another index (decided §10.4).** Objects that `list` ranks, such as a person, are long-lived and have no "active" status. If they did, they would rank through `next`, and `list` would not be needed for them. So the server cannot rank them from the `active` index. The table -gains a second sparse index, keyed by type: +gains a third sparse index — after `active` and `dated` —, keyed by type: | what | name | type | role | |---|---|---|---| @@ -155,10 +197,11 @@ gains a second sparse index, keyed by type: - **Generic.** The fold indexes every live item by its type, so it needs no plugin knowledge of which types `list` ranks. The price is one index write per item, not only for the ranked types. - **Edges.** A derived key on a `list`-ranked type reads that item's `contributes_to` and `mentions` - neighbours, and the counts read their inbound edges. Today the adjacency entries (`.adj#`) are - kept only for `dep`/`parent` edges that touch an active ticket. They are widened to every present - edge of a kind some key names, between live items. That widening is the open design work of the - server cut (§11), together with what it costs per edge write. + neighbours, and the counts read their inbound edges. Corrected in 1c5s: the adjacency entries + (`.adj#`) were always written for every edge add, under both ends; only the `active` INDEX flag is + limited to `dep`/`parent` edges with an active end. Since index revision 3 the entries say what + their edge is and whether it is removed (`next` above), so a `list`-ranked item reads its edges + with one Query on its own prefix, at no extra cost per edge write. - **Archived items.** They are in the index, because they are live. `list` filters them at read, as it does locally. - **It changes the table contract.** The deploying stack (the web package's CDK) must add the GSI @@ -166,6 +209,43 @@ gains a second sparse index, keyed by type: still writes the attribute, which costs nothing. Only the `list` read needs the index. The rollout order is therefore: the stack first, then the read. +**`list` as built (xkr7).** +- **The key.** `nxf_typed` is `#` with the type escaped as a key value is (`%` → + `%25`, `#` → `%23`), so `a` + `b#c` and `a#b` + `c` cannot share a partition; the sort key is the + table's `sk`, so a Query lists a type's items in id order. An item without a type is on no + partition. A type whose partition would pass the 2,048 bytes DynamoDB allows an index key is + refused at fold time, as an op with a sort key too long is. +- **Index revision.** The attribute ships in revision 3 with the self-describing adjacency entries + (1c5s and xkr7 reach the same beta), so revision 3 still makes every older stream not current + and one refold sets both; no revision 4. +- **Where it lives.** The facade's `read::list_reads(cfg, query)` names what a hosted `list` reads + (`ListReads`: the listed types — the query's type filter, else every type —, whether `next`'s + board is needed, and the graph); `nxs_fold_ddb::records::list_board` reads it into a `ListBoard` + (the listed items with the joins a `list` record and its filters carry, `next`'s board, the + `RankingGraph`); `read::list_board_page` and `read::list_board_value` filter, rank and project it + by the code `list_page` runs, and `read::list_board` reads the same board from a store, the + parity oracle. +- **Without a type filter** the server lists every live item: it reads the partition's `items#` + rows (strongly consistent, one Query — paged by DynamoDB at 1 MB —, no index), since the index + lists one type at a time. +- **What a list costs** (review of PR #55, Integrity #1). A `list` that is not ranked orders and + filters by the item rows alone, so `ListReads::select` lets the server keep and cut BEFORE the + joins: status and type are asked of the row, a container or label filter reads the one join it + needs per kept row, and only the page (at most `limit` rows) reads the rest — three Queries a + row, the timestamps and outside parents by `BatchGetItem`. `list --limit 10` costs the rows' + Query and about 33 requests, however many items the stream holds. A RANKED `list` reads every + live, unarchived item of its types in full — three Queries each — because a rank is computed + over the type's whole `list` scope (§12, "The scope of a `list` rank": every live, unarchived item of the type, + whatever the query's status, filter or limit). That cost is the scope's, not the answer's, and it + is not capped: a hard per-call budget is filed as `6j6v.wq9e` (E4 spec §5.1). +- **Paging.** As for the hosted `next`: the store path's page cache is machine-local and a + server's paging is an open decision (15ed), so the store-free `list` is not paged — `limit` cuts + it with the total disclosed, `paginate` and a token are refused with `validation`, and a server + cuts the list itself. +- **`now`.** A `list` key that reads a rank `next` gives takes `next`'s candidates at the query's + `now`, as locally (§12); without one the read answers `validation` where a ranked row needs it, + and `next`'s board is not read for it. + A differential test holds the SQL path and the server path to the same order on generated boards, for `next` and for `list`, as for the lanes today. @@ -222,8 +302,13 @@ the plugin declares (`plugin-custom-fields.md`), as decided in §10.2. - **Per kind:** unit tests over hand-built boards: a value from one neighbour, from several (`min`/`max`), from none (null placement); counts with each filter. - **Chains:** actions by project rank by goal order across three types, and a refused cycle. -- **Differential:** SQL path against `graph::Board` on generated boards with derived keys, as - `fold_differential` does for the lanes. +- **Differential:** SQL path against the server path on generated boards with derived keys, as + `fold_differential` does for the lanes. Built for `next` in 1c5s + (`crates/fold-ddb/tests/derived_differential.rs`): the SQL path, the store-free path over the + store's own board and graph, and the same over the folded tables give one order. Built for + `list` in xkr7 (`crates/fold-ddb/tests/list_differential.rs`): the same three paths on + generated boards of four types with retypes, every key kind and cross-section ranks both ways, + over a set of queries (types, none, container, label, status, sort, limit), and the JSON. - **Determinism:** shuffled edge order gives the same result, and ties fall to the id. - **Bounded cost:** a ratio guard in the style of `xn8s`. A query with derived keys runs the edge pass once, not once per candidate. @@ -241,19 +326,34 @@ the plugin declares (`plugin-custom-fields.md`), as decided in §10.2. 3. **Persons rank by `sum`** (§3.3), equal weights, `weight` left out until data asks for it. Strictly sequential keys were declined: one more mention would always beat any number of open waiting-fors. -4. **A hosted `list` with derived keys uses a second index** (§6, GSI `typed`). Ranking only active +4. **A hosted `list` with derived keys uses a second index** (§6, GSI `typed`; the table's third, + counting `dated`, which this decision did not count). Ranking only active items was declined: the objects `list` ranks have no active status by design. If they had one, they would rank through `next`, and `list` would not be needed. Leaving `list` unranked on the server was declined because the product reads through the hosted path. +5. **On the server a key reads every live neighbour** (owner, 2026-10-11, during 1c5s). The first + text of §6 said a counterpart that is not an active ticket counts as absent; the local build + (§12) reads every live neighbour, closed and archived ones included. The owner chose the local + rule for both paths, so a server and a replica rank alike: the fold keeps self-describing + adjacency entries (index revision 3) and the server reads the neighbours that are not active. + Narrowing the local rule to active neighbours was declined, as was refusing on the server every + key whose value can depend on a neighbour that is not active (it would refuse §8's first order). + A `[ranking.next]` order that reads the rank of a type only `[ranking.list]` orders was refused on + the server until the `typed` index (xkr7) gave it that type's scope; it ranks there since. ## 11. Cuts - **Derived keys, local.** The three key kinds, load-time validation and the type chain over the SQL path. Usable in `next` (after `z9jk`) and `list` (after `n698`). -- **Derived keys on the server, `next`.** `graph::Board` with the extra edges and fields; the - differential test. -- **The `typed` index and a ranked hosted `list`.** The GSI, the widened adjacency, the read, and - the table-contract note for the deploying stack, coordinated with app-foundations. +- **Derived keys on the server, `next`** (1c5s, built). The `RankingGraph` on the store-free + lanes' board, the self-describing adjacency entries (index revision 3), and the differential test + (SQL path = Board path = fold-ddb path on generated boards with closed, archived and deleted + neighbours, mentions and inbound `contributes_to`). +- **The `typed` index and a ranked hosted `list`** (xkr7, built). The GSI, the read + (`ListBoard`, `records::list_board`, `read::list_board_page`), the `[ranking.list]` rank in a + hosted `next`, the differential test, and the table-contract note for the deploying stack + (`nxs-fold-ddb` crate doc, E4 §5.1), coordinated with app-foundations. The widened adjacency + turned out to be 1c5s's self-describing entries; nothing more was needed. - **Shared with `6j6v.q29p`.** The reverse mention lookup and `count = "mentioned_by"` use one implementation. @@ -286,7 +386,10 @@ reading of the text, so a later decision can loosen it without breaking a declar `blocked` and the lanes' `--sort rank`, which compute none. The `[ranking.list]` default order takes them. - **`nulls` on a `count` or `sum`** is refused: a count is never null (§3.2). -- **The store-free lanes** (`read::next_active`) refuse a `[ranking.next]` with derived keys until - the server cut (1c5s) gives them the edges. +- **The store-free lanes** (`read::next_active`) rank a `[ranking.next]` with derived keys from + the board's `RankingGraph` since 1c5s (§6), with the live-neighbour rule above; since xkr7 an + order that reads the rank of a type only `[ranking.list]` orders too, from the graph's listed + items. The store-free `list` (`read::list_board_page`, xkr7) ranks a `[ranking.list]` the same + way. - **TOML.** An inline table may not span lines outside its arrays (TOML 1.0), so a `sum`'s `dir` stands on the line its array closes on. §8 above breaks that line for reading only.