From a442d541f41e80fccff303aa3ec677c3806de235 Mon Sep 17 00:00:00 2001 From: Carsten Koch Date: Sun, 11 Oct 2026 18:29:24 +0200 Subject: [PATCH 1/3] Derived ranking keys on the hosted server for next (nxf 6j6v.1c5s) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit The store-free `next` (read::next_active) now ranks a plugin with `related` / `count` / `sum` keys from a RankingGraph on ActiveBoard, by the same evaluation the store path runs. Owner decision 2026-10-11: a key reads every LIVE neighbour (closed and archived count, deleted do not) on both paths. - facade: ActiveBoard::ranking, RankingReads, RankingGraph, Neighbour, next_ranking_reads, with_ranking (the parity oracle); derived.rs split into plan / context fill (store or graph) / finish. A [ranking.next] order reading the rank of a type only [ranking.list] orders is refused on the store-free path until xkr7. - fold-ddb: .adj# entries copy from_id/to_id/kind/lamport/site and a removed flag, kept on edge removes; INDEX_REVISION 2 -> 3 (old streams NotCurrent until refolded); records::with_ranking reads the graph. Table contract unchanged (no new GSI, no new key attribute). - core: EdgeKind and graph::Edge derive Ord. - tests: derived_differential (SQL = Board = fold-ddb on generated boards, determinism, request-count guard, revision-2 refold), facade seam tests for live neighbours and the list-rank refusal. - spec §6/§10.5/§11/§12 amended. Co-Authored-By: Claude Opus 5.5 Claude-Session: https://claude.ai/code/session_016q5uMUkZJQW4fckJoA3FME --- Cargo.lock | 1 + changes/1c5s-server-derived-keys.md | 8 + crates/core/src/graph.rs | 3 +- crates/core/src/model.rs | 2 +- crates/facade/src/read.rs | 210 +++++++- crates/facade/src/read/derived.rs | 405 ++++++++++++--- crates/facade/tests/derived_ranking.rs | 207 +++++++- crates/fold-ddb/Cargo.toml | 3 + crates/fold-ddb/src/board.rs | 77 ++- crates/fold-ddb/src/fold.rs | 8 +- crates/fold-ddb/src/lib.rs | 9 +- crates/fold-ddb/src/records.rs | 176 ++++++- crates/fold-ddb/tests/derived_differential.rs | 483 ++++++++++++++++++ docs/specs/derived-ranking-keys.md | 80 ++- 14 files changed, 1529 insertions(+), 143 deletions(-) create mode 100644 changes/1c5s-server-derived-keys.md create mode 100644 crates/fold-ddb/tests/derived_differential.rs diff --git a/Cargo.lock b/Cargo.lock index b695c61..e10c7f3 100644 --- a/Cargo.lock +++ b/Cargo.lock @@ -2442,6 +2442,7 @@ version = "0.209.0" dependencies = [ "aws-sdk-dynamodb", "futures-util", + "inventory", "nexus-chat", "nexus-flow-core", "nexus-flow-facade", diff --git a/changes/1c5s-server-derived-keys.md b/changes/1c5s-server-derived-keys.md new file mode 100644 index 0000000..1d63ec8 --- /dev/null +++ b/changes/1c5s-server-derived-keys.md @@ -0,0 +1,8 @@ +--- +type: added +facade: changed +--- +[en] +A hosted board now ranks `next` by derived ranking keys (`related`, `count`, `sum`) exactly as a local replica does. The store-free `next` (`read::next_active`, `next_active_value`, `next_active_filtered`) ranks a plugin with such keys from a new `ActiveBoard::ranking` graph: the edges around the active tickets and the live neighbours they reach, closed and archived ones included, deleted ones not. A server fills it with `nxs_fold_ddb::records::with_ranking` for what `read::next_ranking_reads(cfg)` names; `read::with_ranking` reads the same graph from a store. Without the graph, `next_active` still refuses such a plugin. One case stays refused on the store-free path: a `[ranking.next]` order that reads the rank of a type only `[ranking.list]` orders — answer that `next` from a store. Everything is additive: `ActiveBoard` gains a field (it is `#[non_exhaustive]`), and `RankingReads`, `RankingGraph`, `Neighbour`, `next_ranking_reads` and `with_ranking` are new. For a server's operators: the fold's index revision goes from 2 to 3, so every hosted stream is not current and is folded again on its next run; until then its records read answers `NotCurrent`. The table contract is unchanged: no new index and no new key attribute, only more attributes on the existing `.adj#` adjacency entries. +[de] +Ein gehostetes Brett rankt `next` jetzt nach abgeleiteten Ranking-Schlüsseln (`related`, `count`, `sum`) genau so wie ein lokales Replikat. Das speicherfreie `next` (`read::next_active`, `next_active_value`, `next_active_filtered`) rankt ein Plugin mit solchen Schlüsseln aus einem neuen Graphen `ActiveBoard::ranking`: den Kanten um die aktiven Tickets und den lebenden Nachbarn, die sie erreichen, geschlossene und archivierte eingeschlossen, gelöschte nicht. Ein Server füllt ihn mit `nxs_fold_ddb::records::with_ranking` für das, was `read::next_ranking_reads(cfg)` nennt; `read::with_ranking` liest denselben Graphen aus einem Speicher. Ohne den Graphen lehnt `next_active` ein solches Plugin weiterhin ab. Ein Fall bleibt auf dem speicherfreien Weg abgelehnt: eine Reihenfolge in `[ranking.next]`, die den Rang eines Typs liest, den nur `[ranking.list]` ordnet — dieses `next` beantwortet ein Speicher. Alles ist additiv: `ActiveBoard` bekommt ein Feld (es ist `#[non_exhaustive]`), und `RankingReads`, `RankingGraph`, `Neighbour`, `next_ranking_reads` und `with_ranking` sind neu. Für den Betrieb eines Servers: Die Index-Revision des Folds steigt von 2 auf 3, daher ist jeder gehostete Stream nicht aktuell und wird bei seinem nächsten Lauf neu gefaltet; bis dahin antwortet sein Records-Lesen mit `NotCurrent`. Der Tabellenvertrag bleibt unverändert: kein neuer Index und kein neues Schlüsselattribut, nur mehr Attribute an den bestehenden `.adj#`-Adjazenzeinträgen. diff --git a/crates/core/src/graph.rs b/crates/core/src/graph.rs index dca4ce6..34b8f84 100644 --- a/crates/core/src/graph.rs +++ b/crates/core/src/graph.rs @@ -30,7 +30,8 @@ pub struct Ticket { } /// A present edge, `from` → `to` (a `dep` points at its blocker, a `parent` edge at the parent). -#[derive(Debug, Clone, PartialEq, Eq)] +/// Ordered by `from`, `to`, `kind`, so a set of edges has one order whatever order they came in. +#[derive(Debug, Clone, PartialEq, Eq, PartialOrd, Ord)] pub struct Edge { pub from: String, pub to: String, diff --git a/crates/core/src/model.rs b/crates/core/src/model.rs index e00fb57..5beca2d 100644 --- a/crates/core/src/model.rs +++ b/crates/core/src/model.rs @@ -48,7 +48,7 @@ impl Status { /// the OR-set replacement for the former single-parent `belongs_to` LWW register. The substrate is /// n:m-capable like every other edge; single-parent is enforced at the write seam during ②a and /// relaxed in ②c. The single current parent is projected by the `present_parent` view. -#[derive(Debug, Clone, Copy, PartialEq, Eq, Hash)] +#[derive(Debug, Clone, Copy, PartialEq, Eq, PartialOrd, Ord, Hash)] pub enum EdgeKind { Dep, ContributesTo, diff --git a/crates/facade/src/read.rs b/crates/facade/src/read.rs index 47d775b..7784967 100644 --- a/crates/facade/src/read.rs +++ b/crates/facade/src/read.rs @@ -24,7 +24,7 @@ use crate::plugin::{Dir, Nulls, PluginConfig, RankKey, RankSpec, Verb}; use crate::record::{item_value, items_value, raw_field}; use nexus_flow_core::derive; use nexus_flow_core::graph::Board; -use nexus_flow_core::model::{ItemRow, LinkWeight, ThreadLink}; +use nexus_flow_core::model::{EdgeKind, ItemRow, LinkWeight, ThreadLink}; use nexus_flow_core::store::Store; use serde_json::{json, Value}; use std::collections::{BTreeMap, BTreeSet, HashMap, HashSet}; @@ -2734,6 +2734,11 @@ pub struct ActiveBoard { pub board: Board, /// By id. pub tickets: BTreeMap, + /// The graph the derived ranking keys of `[ranking.next]` read (6j6v.1c5s): `None` until a + /// reader fills it — [`with_ranking`] from a store, `nxs_fold_ddb::records::with_ranking` from + /// the folded tables — for what [`next_ranking_reads`] names. A plugin without derived `next` + /// keys never needs it; [`next_active`] refuses one with such keys while it is missing. + pub ranking: Option, } impl ActiveBoard { @@ -2745,6 +2750,7 @@ impl ActiveBoard { .into_iter() .map(|t| (t.item.id.clone(), t)) .collect(), + ranking: None, } } @@ -2780,6 +2786,163 @@ impl ActiveBoard { } } +/// What the derived ranking keys of a `[ranking.next]` read beyond the active tickets' records +/// (6j6v.1c5s) — what a reader of an [`ActiveBoard`] puts in its [`RankingGraph`]. +/// [`next_ranking_reads`] gives it for a plugin. +/// +/// `#[non_exhaustive]`: start from `RankingReads::default()` and set the fields. +#[derive(Debug, Clone, Default, PartialEq, Eq)] +#[non_exhaustive] +pub struct RankingReads { + /// The edge kinds the keys follow or count; `mentions` too when a key counts `mentioned_by`. + pub kinds: BTreeSet, + /// A count filters on a label: the neighbours' labels. + pub labels: bool, + /// An `of` reads a custom field: the neighbours' custom values. + pub custom: bool, +} + +impl RankingReads { + /// Whether a graph read for `self` holds everything a graph read for `needed` holds. + pub fn covers(&self, needed: &RankingReads) -> bool { + needed.kinds.is_subset(&self.kinds) + && (self.labels || !needed.labels) + && (self.custom || !needed.custom) + } +} + +/// The graph around a board's active tickets that the derived ranking keys read (6j6v.1c5s, +/// docs/specs/derived-ranking-keys.md §6): the input [`next_active`] ranks a `[ranking.next]` with +/// `related` / `count` / `sum` keys from, so a server ranks exactly as [`next`] does over a store. +/// +/// - `edges`: every present edge of a kind in `reads.kinds` with at least one ACTIVE end (an +/// [`ActiveBoard::tickets`] entry) and both ends LIVE (not deleted, not missing). A `parent` edge +/// is the `present_parent` projection — a child's one current parent, the one `belongs_to` reads — +/// so a child re-parented concurrently is under one parent only. Distinct, ordered. +/// - `neighbours`: the other ends of `edges` that are not active tickets — closed and archived +/// items included, since a key reads every LIVE neighbour (owner, 2026-10-11) — each with its row +/// (`belongs_to` filled as the store fills it), its labels when `reads.labels` and its non-empty +/// custom values when `reads.custom`, else empty. +/// +/// `#[non_exhaustive]`: build it with [`RankingGraph::new`] and set the fields. +#[derive(Debug, Clone, Default, PartialEq, Eq)] +#[non_exhaustive] +pub struct RankingGraph { + /// What the graph was read for. + pub reads: RankingReads, + pub edges: BTreeSet, + /// By id. + pub neighbours: BTreeMap, +} + +impl RankingGraph { + /// An empty graph read for `reads`. + pub fn new(reads: RankingReads) -> RankingGraph { + RankingGraph { + reads, + ..RankingGraph::default() + } + } +} + +/// A live item next to an active ticket that is not itself active: what a derived ranking key reads +/// of it ([`RankingGraph::neighbours`]). +/// +/// `#[non_exhaustive]`: build it with [`Neighbour::new`] and set the fields. +#[derive(Debug, Clone, PartialEq, Eq)] +#[non_exhaustive] +pub struct Neighbour { + pub item: ItemRow, + /// Present labels, sorted and distinct, as `present_labels`. + pub labels: Vec, + /// Every non-empty custom value, as the store reads them in bulk. + pub custom: BTreeMap, +} + +impl Neighbour { + /// A neighbour with no labels or custom values read. + pub fn new(item: ItemRow) -> Neighbour { + Neighbour { + item, + labels: Vec::new(), + custom: BTreeMap::new(), + } + } +} + +/// What the store-free `next` of `cfg` reads beyond the active tickets' records — the +/// [`RankingReads`] a reader fills [`ActiveBoard::ranking`] for — or `None` when `[ranking.next]` +/// declares no derived ranking key (6j6v.1c5s). It does not depend on the board. +/// +/// `validation` for a `[ranking.next]` that reads, directly or down its type chain, the rank of a +/// type only `[ranking.list]` orders (docs/specs/derived-ranking-keys.md §6): that rank is computed +/// over the type's live items, and a board of active tickets does not hold them. The store path +/// ([`next`]) ranks such a plugin. +pub fn next_ranking_reads(cfg: &PluginConfig) -> Result> { + derived::reads_for_next(cfg) +} + +/// `board` with its [`RankingGraph`] for `reads` read from `store` (6j6v.1c5s) — the parity oracle +/// for `nxs_fold_ddb::records::with_ranking`, as [`active_board`] is for the records. One edge pass +/// ([`Store::present_edges_of_kinds`]), one bulk read of the neighbours' rows, and one of their +/// labels and of their custom values when `reads` asks for them. +pub fn with_ranking( + store: &Store, + mut board: ActiveBoard, + reads: &RankingReads, +) -> Result { + let kinds: Vec = reads.kinds.iter().copied().collect(); + let active = |id: &str| board.tickets.contains_key(id); + let touching: Vec<(String, String, EdgeKind)> = store + .present_edges_of_kinds(&kinds)? + .into_iter() + .filter(|(from, to, _)| active(from) || active(to)) + .collect(); + let others: BTreeSet<&str> = touching + .iter() + .flat_map(|(from, to, _)| [from.as_str(), to.as_str()]) + .filter(|id| !active(id)) + .collect(); + let others: Vec<&str> = others.into_iter().collect(); + let live: BTreeMap = store + .get_items(&others)? + .into_iter() + .filter(|i| i.deleted.as_deref() != Some("1")) + .map(|i| (i.id.clone(), i)) + .collect(); + let edges: BTreeSet = touching + .into_iter() + .filter(|(from, to, _)| { + (active(from) || live.contains_key(from)) && (active(to) || live.contains_key(to)) + }) + .map(|(from, to, kind)| nexus_flow_core::graph::Edge { from, to, kind }) + .collect(); + let ids: Vec<&str> = live.keys().map(String::as_str).collect(); + let mut labels = if reads.labels { + store.labels_of_bulk(&ids)? + } else { + BTreeMap::new() + }; + let mut custom = if reads.custom { + store.custom_fields_of_bulk(&ids)? + } else { + BTreeMap::new() + }; + let mut graph = RankingGraph::new(reads.clone()); + graph.edges = edges; + graph.neighbours = live + .into_iter() + .map(|(id, item)| { + let mut n = Neighbour::new(item); + n.labels = labels.remove(&id).unwrap_or_default(); + n.custom = custom.remove(&id).unwrap_or_default(); + (id, n) + }) + .collect(); + board.ranking = Some(graph); + Ok(board) +} + /// The store path's input as an [`ActiveBoard`]: the store's selection ([`graph::select`]) and, for /// every active ticket, its row and joins, each read in bulk. What [`next_active`] and the other /// store-free lanes answer from it is what [`next`] and its siblings answer from the store. @@ -2823,21 +2986,44 @@ pub fn active_board(store: &Store) -> Result { /// /// [`Engine::next`]: crate::engine::Engine::next /// -/// A `[ranking.next]` with derived ranking keys (6j6v.x0wf) is refused with `validation`: an -/// [`ActiveBoard`] holds neither the edges nor the scopes those keys read, and ranking without them -/// would order apart from the store path in silence. The server path is its own cut (6j6v.1c5s). +/// A `[ranking.next]` with derived ranking keys (6j6v.x0wf) is ranked from the board's +/// [`RankingGraph`] (6j6v.1c5s), by the same evaluation the store path runs. It is refused with +/// `validation` when the board carries no graph, or one read for less than +/// [`next_ranking_reads`] names — ranking without those edges would order apart from the store path +/// in silence — and when [`next_ranking_reads`] refuses the plugin. pub fn next_active(cfg: &PluginConfig, board: &ActiveBoard, now: &str) -> Result> { crate::validate::iso_date(now)?; - if cfg.ranking.has_derived(Verb::Next) { - return Err(NxfError::validation(format!( - "plugin '{}' ranks `next` by derived ranking keys, which the store-free lanes do not \ - compute yet — answer `next` from a store", - cfg.name - ))); - } + let graph = match next_ranking_reads(cfg)? { + None => None, + Some(needed) => Some( + board + .ranking + .as_ref() + .filter(|g| g.reads.covers(&needed)) + .ok_or_else(|| { + NxfError::validation(format!( + "plugin '{}' ranks `next` by derived ranking keys, and the board was read \ + without the graph they read — fill `ActiveBoard::ranking` for \ + `next_ranking_reads` (`read::with_ranking` from a store, \ + `records::with_ranking` from the folded tables)", + cfg.name + )) + })?, + ), + }; let candidates = board.board.lanes(now).candidates; let items = board.items(candidates.iter().map(|c| &c.id)); - Ok(next_lane(cfg, candidates, items, DEFAULT_SORT_NEXT, None)) + let derived = match graph { + Some(graph) => derived::compute_active(cfg, board, graph, &items)?, + None => None, + }; + Ok(next_lane( + cfg, + candidates, + items, + DEFAULT_SORT_NEXT, + derived.as_ref(), + )) } /// [`next_active`], narrowed by `filter` (6j6v.15ed): the rows [`next_filtered`] gives over a store diff --git a/crates/facade/src/read/derived.rs b/crates/facade/src/read/derived.rs index 86e3e3a..9906834 100644 --- a/crates/facade/src/read/derived.rs +++ b/crates/facade/src/read/derived.rs @@ -19,10 +19,17 @@ //! //! So a query costs O(E + N log N) in the relevant edges and the items in scope, and a plugin //! that declares no derived key pays nothing: [`compute`] returns `None` before any read. +//! +//! **Without a store** (6j6v.1c5s). The store-free `next` ([`super::next_active`]) ranks a board a +//! server read: [`compute_active`] takes steps 1 and 2 from the board's [`RankingGraph`] — the +//! edges, already projected and filtered to live ends, and the live neighbours' rows, labels and +//! custom values — and runs steps 3 and 4 on the same [`Context`], so the two paths cannot evaluate +//! a key apart. A neighbour counts when it is LIVE, closed and archived ones included, on both +//! paths (spec §6, owner 2026-10-11). What the graph must hold is [`reads_for_next`]. -use super::{lanes, mentioners_of, resolve_items}; +use super::{lanes, mentioners_of, resolve_items, ActiveBoard, RankingGraph, RankingReads}; use crate::error::{NxfError, Result}; -use crate::plugin::{Dir, Nulls, Pick, PluginConfig, RankKey, RankOf, Verb}; +use crate::plugin::{Dir, Nulls, Pick, PluginConfig, RankKey, RankOf, RankSpec, Verb}; use crate::record::{raw_field, RANKABLE_FIELDS}; use nexus_flow_core::model::{EdgeKind, ItemRow}; use nexus_flow_core::store::Store; @@ -131,20 +138,20 @@ pub(super) fn list_scope(store: &Store) -> Result> { .collect()) } -/// The values the derived keys of `verb`'s policy rank `ranked` by — `None`, before any read, -/// when the policy has no derived key. `validation` when a `list` key reads the rank of a type -/// `next` ranks and the caller passed neither `next`'s candidates nor a reference time. -pub(super) fn compute( - cfg: &PluginConfig, - store: &Store, - verb: Verb, - ranked: &[ItemRow], - scopes: Scopes<'_>, -) -> Result> { +/// The derived side of one query, decided before any read: the type chain the ranked rows' orders +/// read the rank of (leaves first), and what the derived keys in play read. +struct Plan<'c> { + spec: &'c RankSpec, + chain: Vec, + needs: Needs, +} + +/// The [`Plan`] for ranking `ranked` under `verb` — `None` when the policy has no derived key. +fn plan<'c>(cfg: &'c PluginConfig, verb: Verb, ranked: &[ItemRow]) -> Option> { let ranking = &cfg.ranking; let spec = ranking.spec(verb); if !spec.has_derived() { - return Ok(None); + return None; } // The chain, leaves first: the orders the ranked rows' orders read the rank of, transitively. @@ -158,10 +165,77 @@ pub(super) fn compute( } } + // Every derived key in play. + let mut keys: Vec<&RankKey> = Vec::new(); + for ty in &ranked_types { + keys.extend(spec.order_for(*ty).iter().filter(|k| k.is_derived())); + } + for (v, ty) in &chain { + keys.extend( + ranking.spec(*v).by_type[ty] + .order + .iter() + .filter(|k| k.is_derived()), + ); + } + let needs = Needs::of(&keys); + Some(Plan { spec, chain, needs }) +} + +impl Plan<'_> { + /// The rows each order of the chain ranks: its type's rows in the scope of its verb. + fn node_rows<'r>(&self, next: &'r [ItemRow], list: &'r [ItemRow]) -> Vec> { + self.chain + .iter() + .map(|(v, ty)| { + let scope = match v { + Verb::Next => next, + Verb::List => list, + }; + scope + .iter() + .filter(|i| i.item_type.as_deref() == Some(ty.as_str())) + .collect() + }) + .collect() + } + + /// Whether some order of the chain is a `[ranking.list]` order. + fn reads_list(&self) -> bool { + self.chain.iter().any(|(v, _)| *v == Verb::List) + } +} + +/// Every row a key is evaluated on: the ranked rows and the rows of the chain's scopes. +fn subjects<'a>( + ranked: &'a [ItemRow], + node_rows: &[Vec<&'a ItemRow>], +) -> HashMap<&'a str, &'a ItemRow> { + let mut subjects: HashMap<&str, &ItemRow> = HashMap::new(); + for row in ranked.iter().chain(node_rows.iter().flatten().copied()) { + subjects.insert(row.id.as_str(), row); + } + subjects +} + +/// The values the derived keys of `verb`'s policy rank `ranked` by — `None`, before any read, +/// when the policy has no derived key. `validation` when a `list` key reads the rank of a type +/// `next` ranks and the caller passed neither `next`'s candidates nor a reference time. +pub(super) fn compute( + cfg: &PluginConfig, + store: &Store, + verb: Verb, + ranked: &[ItemRow], + scopes: Scopes<'_>, +) -> Result> { + let Some(plan) = plan(cfg, verb, ranked) else { + return Ok(None); + }; + // The scopes the chain ranks over, read only when the caller does not hold them. let next_owned; let next_rows: &[ItemRow] = match scopes.next { - _ if !chain.iter().any(|(v, _)| *v == Verb::Next) => &[], + _ if !plan.chain.iter().any(|(v, _)| *v == Verb::Next) => &[], Some(rows) => rows, None => { let now = scopes.now.ok_or_else(|| { @@ -178,89 +252,115 @@ pub(super) fn compute( }; let list_owned; let list_rows: &[ItemRow] = match scopes.list { - _ if !chain.iter().any(|(v, _)| *v == Verb::List) => &[], + _ if !plan.reads_list() => &[], Some(rows) => rows, None => { list_owned = list_scope(store)?; &list_owned } }; - let node_rows: Vec> = chain - .iter() - .map(|(v, ty)| { - let scope = match v { - Verb::Next => next_rows, - Verb::List => list_rows, - }; - scope - .iter() - .filter(|i| i.item_type.as_deref() == Some(ty.as_str())) - .collect() - }) - .collect(); + let node_rows = plan.node_rows(next_rows, list_rows); + let mut ctx = Context::new(cfg, subjects(ranked, &node_rows)); + ctx.read_edges(store, &plan.needs)?; + Ok(Some(ctx.finish(&plan, verb, ranked, &node_rows))) +} - // Every row a key is evaluated on, and every derived key in play. - let mut subjects: HashMap<&str, &ItemRow> = HashMap::new(); - for row in ranked.iter().chain(node_rows.iter().flatten().copied()) { - subjects.insert(row.id.as_str(), row); +/// [`compute`] for the store-free `next` (6j6v.1c5s): the values the derived keys of +/// `[ranking.next]` rank `candidates` — every candidate of the board — by, read from the +/// [`RankingGraph`] a reader filled instead of from a store. The candidates are `next`'s scope for +/// "rank", as on the store path. `None` when the policy has no derived key. +/// +/// The same [`Context`] evaluates the keys as on the store path; only where its edges, rows, +/// labels and custom values come from differs ([`Context::read_graph`]). A chain that reaches a +/// `[ranking.list]` order is refused before this is called ([`reads_for_next`]); it answers +/// `validation` here too rather than rank without that scope. +pub(super) fn compute_active( + cfg: &PluginConfig, + board: &ActiveBoard, + graph: &RankingGraph, + candidates: &[ItemRow], +) -> Result> { + let Some(plan) = plan(cfg, Verb::Next, candidates) else { + return Ok(None); + }; + if plan.reads_list() { + return Err(list_scope_refusal(cfg)); } - let mut keys: Vec<&RankKey> = Vec::new(); - for ty in &ranked_types { - keys.extend(spec.order_for(*ty).iter().filter(|k| k.is_derived())); + let node_rows = plan.node_rows(candidates, &[]); + let mut ctx = Context::new(cfg, subjects(candidates, &node_rows)); + ctx.read_graph(board, graph, &plan.needs); + Ok(Some(ctx.finish(&plan, Verb::Next, candidates, &node_rows))) +} + +/// What the store-free `next` of `cfg` reads beyond the active tickets' records (6j6v.1c5s): +/// `None` when `[ranking.next]` declares no derived key. Gathered over every order of the section, +/// so it does not depend on which types a board holds. +/// +/// `validation` when an order of `[ranking.next]` reads — directly or down its type chain — the +/// rank of a type only `[ranking.list]` orders (spec §12, first point): that rank is computed over +/// the live items of the type, and a server reads only the active ones (owner, 2026-10-11). +pub(super) fn reads_for_next(cfg: &PluginConfig) -> Result> { + let ranking = &cfg.ranking; + let spec = ranking.spec(Verb::Next); + if !spec.has_derived() { + return Ok(None); } - for (v, ty) in &chain { - keys.extend( - ranking.spec(*v).by_type[ty] - .order - .iter() - .filter(|k| k.is_derived()), - ); + let mut chain: Vec = Vec::new(); + let mut seen: BTreeSet = BTreeSet::new(); + for ty in spec.by_type.keys() { + post_order(cfg, (Verb::Next, ty.clone()), &mut seen, &mut chain); } - + if chain.iter().any(|(v, _)| *v == Verb::List) { + return Err(list_scope_refusal(cfg)); + } + let keys: Vec<&RankKey> = spec + .order + .iter() + .chain(spec.by_type.values().flat_map(|o| o.order.iter())) + .filter(|k| k.is_derived()) + .collect(); let needs = Needs::of(&keys); - let mut ctx = Context { - cfg, - subjects, - out: Adjacency::new(), - inb: Adjacency::new(), - mentioners: BTreeMap::new(), - neighbours: HashMap::new(), - labels: BTreeMap::new(), - custom: BTreeMap::new(), - ranks: HashMap::new(), - }; - ctx.read_edges(store, &needs)?; - - // Bottom-up: each order of the chain ranks its scope once; its dependents look the rank up. - for (node, rows) in chain.iter().zip(&node_rows) { - let keys = &ranking.spec(node.0).by_type[&node.1].order; - let values: Vec> = rows.iter().map(|r| ctx.values(keys, node.0, r)).collect(); - let mut order: Vec = (0..rows.len()).collect(); - let cmp = |a: usize, b: usize| { - super::keys_cmp_values(rows[a], rows[b], keys, Some(&values[a]), Some(&values[b])) - }; - order.sort_by(|a, b| cmp(*a, *b).then_with(|| rows[*a].id.cmp(&rows[*b].id))); - // A DENSE rank over the declared keys (spec §4): rows equal on every key share a rank, and - // the next distinct row takes the next integer. The id orders the sort, never the rank. - let mut ranks: HashMap = HashMap::with_capacity(rows.len()); - let mut rank = 0_i64; - for (n, &i) in order.iter().enumerate() { - if n > 0 && cmp(order[n - 1], i) != Ordering::Equal { - rank += 1; - } - ranks.insert(rows[i].id.clone(), rank); - } - ctx.ranks.insert(node.clone(), ranks); + let mut reads = RankingReads::default(); + reads.kinds.extend(needs.kinds.iter().copied()); + if needs.mentioned_by { + reads.kinds.insert(EdgeKind::Mentions); } + reads.labels = needs.labels; + reads.custom = needs.custom; + Ok(Some(reads)) +} - let values = ranked - .iter() - .map(|r| { - let keys = spec.order_for(r.item_type.as_deref()); - (r.id.clone(), ctx.values(keys, verb, r)) +/// The refusal of a `[ranking.next]` whose type chain reaches a `[ranking.list]` order, on the +/// store-free path. +fn list_scope_refusal(cfg: &PluginConfig) -> NxfError { + let ranking = &cfg.ranking; + let reached: Vec = ranking + .spec(Verb::Next) + .by_type + .keys() + .flat_map(|ty| { + let mut chain = Vec::new(); + post_order( + cfg, + (Verb::Next, ty.clone()), + &mut BTreeSet::new(), + &mut chain, + ); + chain + .into_iter() + .filter(|(v, _)| *v == Verb::List) + .map(move |(_, list_ty)| { + format!("[ranking.next.{ty}] reads the rank of '{list_ty}'") + }) }) .collect(); - Ok(Some(Derived { values })) + NxfError::validation(format!( + "plugin '{}' ranks `next` by the rank of a type only [ranking.list] orders ({}): that rank \ + is computed over the type's live items, which the store-free lanes do not hold — answer \ + `next` from a store", + cfg.name, + reached.join("; ") + )) } /// Depth-first post-order over the type chain: every order a node reads the rank of lands before @@ -381,7 +481,146 @@ struct Context<'a> { ranks: HashMap>, } -impl Context<'_> { +impl<'a> Context<'a> { + /// A context over `subjects`, nothing read yet. + fn new(cfg: &'a PluginConfig, subjects: HashMap<&'a str, &'a ItemRow>) -> Context<'a> { + Context { + cfg, + subjects, + out: Adjacency::new(), + inb: Adjacency::new(), + mentioners: BTreeMap::new(), + neighbours: HashMap::new(), + labels: BTreeMap::new(), + custom: BTreeMap::new(), + ranks: HashMap::new(), + } + } + + /// Rank the chain bottom-up — each order its scope once, its dependents look the rank up — and + /// then the ranked rows' values. + fn finish( + mut self, + plan: &Plan<'_>, + verb: Verb, + ranked: &[ItemRow], + node_rows: &[Vec<&ItemRow>], + ) -> Derived { + let ranking = &self.cfg.ranking; + for (node, rows) in plan.chain.iter().zip(node_rows) { + let keys = &ranking.spec(node.0).by_type[&node.1].order; + let values: Vec> = + rows.iter().map(|r| self.values(keys, node.0, r)).collect(); + let mut order: Vec = (0..rows.len()).collect(); + let cmp = |a: usize, b: usize| { + super::keys_cmp_values(rows[a], rows[b], keys, Some(&values[a]), Some(&values[b])) + }; + order.sort_by(|a, b| cmp(*a, *b).then_with(|| rows[*a].id.cmp(&rows[*b].id))); + // A DENSE rank over the declared keys (spec §4): rows equal on every key share a rank, + // and the next distinct row takes the next integer. The id orders the sort, never the + // rank. + let mut ranks: HashMap = HashMap::with_capacity(rows.len()); + let mut rank = 0_i64; + for (n, &i) in order.iter().enumerate() { + if n > 0 && cmp(order[n - 1], i) != Ordering::Equal { + rank += 1; + } + ranks.insert(rows[i].id.clone(), rank); + } + self.ranks.insert(node.clone(), ranks); + } + let values = ranked + .iter() + .map(|r| { + let keys = plan.spec.order_for(r.item_type.as_deref()); + (r.id.clone(), self.values(keys, verb, r)) + }) + .collect(); + Derived { values } + } + + /// The reads from a [`RankingGraph`] instead of a store (6j6v.1c5s): the same adjacency, reverse + /// mentions, neighbour rows, labels and custom values [`read_edges`](Self::read_edges) reads, + /// taken from the graph's edges and from the board's tickets and the graph's neighbours. The + /// graph holds every edge of the kinds read that touches an active ticket with both ends live, + /// `parent` as the `present_parent` projection — the edges the store's pass yields for subjects + /// that are all active tickets, as `next`'s candidates are. + fn read_graph(&mut self, board: &ActiveBoard, graph: &RankingGraph, needs: &Needs) { + let mut mention_sources: BTreeMap> = BTreeMap::new(); + for e in &graph.edges { + if needs.kinds.contains(&e.kind) { + if self.subjects.contains_key(e.to.as_str()) { + self.inb + .entry((e.kind, e.to.clone())) + .or_default() + .push(e.from.clone()); + } + if self.subjects.contains_key(e.from.as_str()) { + self.out + .entry((e.kind, e.from.clone())) + .or_default() + .push(e.to.clone()); + } + } + if needs.mentioned_by + && e.kind == EdgeKind::Mentions + && self.subjects.contains_key(e.to.as_str()) + { + mention_sources + .entry(e.to.clone()) + .or_default() + .insert(e.from.as_str()); + } + } + // A row of the board: an active ticket's, or a live neighbour's. + let board_row = |id: &str| -> Option<&ItemRow> { + board + .tickets + .get(id) + .map(|t| &t.item) + .or_else(|| graph.neighbours.get(id).map(|n| &n.item)) + }; + for (target, froms) in mention_sources { + let rows: Vec = froms + .into_iter() + .filter_map(|f| board_row(f).cloned()) + .collect(); + if !rows.is_empty() { + self.mentioners.insert(target, rows); + } + } + let missing: BTreeSet<&str> = self + .out + .values() + .chain(self.inb.values()) + .flatten() + .map(String::as_str) + .filter(|id| !self.subjects.contains_key(id)) + .collect(); + self.neighbours = missing + .into_iter() + .filter_map(|id| board_row(id).map(|r| (id.to_string(), r.clone()))) + .collect(); + if needs.labels || needs.custom { + let mut ids: BTreeSet<&str> = self.neighbours.keys().map(String::as_str).collect(); + ids.extend(self.subjects.keys().copied()); + ids.extend(self.mentioners.values().flatten().map(|i| i.id.as_str())); + for id in ids { + let (labels, custom) = match (board.tickets.get(id), graph.neighbours.get(id)) { + (Some(t), _) => (&t.labels, &t.custom), + (None, Some(n)) => (&n.labels, &n.custom), + (None, None) => continue, + }; + if needs.labels && !labels.is_empty() { + self.labels.insert(id.to_string(), labels.clone()); + } + if needs.custom && !custom.is_empty() { + self.custom.insert(id.to_string(), custom.clone()); + } + } + } + } + /// The reads: the edge pass, the reverse mentions, the neighbours' rows, labels and custom /// fields — each ONE bulk read (per 500 ids where the store chunks), whatever the row count. fn read_edges(&mut self, store: &Store, needs: &Needs) -> Result<()> { diff --git a/crates/facade/tests/derived_ranking.rs b/crates/facade/tests/derived_ranking.rs index ae831c4..cdb778c 100644 --- a/crates/facade/tests/derived_ranking.rs +++ b/crates/facade/tests/derived_ranking.rs @@ -359,8 +359,37 @@ fn the_rank_follows_the_board_on_the_next_query() { } #[test] -fn the_store_free_lanes_refuse_a_plugin_with_derived_next_keys() { - // The server path is its own cut (1c5s); ranking without the edges would differ in silence. +fn the_store_free_next_ranks_from_the_ranking_graph_as_the_store_does() { + // 6j6v.1c5s: the board a server reads carries the graph the keys read, and the store-free + // `next` ranks from it exactly as the store path does — the spec's three-type chain. + let tmp = workspace("derived-rank-fixture"); + spec_board(tmp.path()); + let cfg = plugin::load("derived-rank-fixture").unwrap(); + let s = writer(tmp.path()); + let reads = read::next_ranking_reads(&cfg) + .unwrap() + .expect("derived next keys"); + assert_eq!( + reads.kinds.iter().copied().collect::>(), + [EdgeKind::ContributesTo, EdgeKind::Parent], + "the edges the next orders follow or count" + ); + assert!(reads.custom && !reads.labels, "{reads:?}"); + let board = read::with_ranking(&s, read::active_board(&s).unwrap(), &reads).unwrap(); + let got = read::next_active(&cfg, &board, NOW).unwrap(); + assert_eq!(ids(&got), ids(&read::next(&cfg, &s, NOW, None).unwrap())); + assert_eq!(of_type(&got, "project"), PROJECTS); + assert_eq!(of_type(&got, "action"), ACTIONS); + assert_eq!( + read::next_active_value(&cfg, &board, NOW).unwrap(), + read::next_to_value_with_custom(&cfg, &s, &read::next(&cfg, &s, NOW, None).unwrap()) + .unwrap() + ); +} + +#[test] +fn the_store_free_next_refuses_a_board_read_without_the_graph_its_keys_read() { + // Ranking without the edges would order apart from the store path in silence. let tmp = workspace("derived-rank-fixture"); spec_board(tmp.path()); let cfg = plugin::load("derived-rank-fixture").unwrap(); @@ -369,6 +398,180 @@ fn the_store_free_lanes_refuse_a_plugin_with_derived_next_keys() { let err = read::next_active(&cfg, &board, NOW).unwrap_err(); assert_eq!(err.kind, ErrorKind::Validation); assert!(err.msg.contains("derived ranking keys"), "{}", err.msg); + // A graph read for less than the keys need is refused too: here without the custom values + // the goals' `order` lives in. + let mut reads = read::next_ranking_reads(&cfg).unwrap().unwrap(); + reads.custom = false; + let short = read::with_ranking(&s, read::active_board(&s).unwrap(), &reads).unwrap(); + let err = read::next_active(&cfg, &short, NOW).unwrap_err(); + assert_eq!(err.kind, ErrorKind::Validation); + // `blocked` and `deferred` rank by no derived key and need no graph. + read::blocked_active(&cfg, &board); + read::deferred_active(&cfg, &board, NOW).unwrap(); +} + +/// A plugin ranking `next` by its neighbours with no status filter, so closed, archived and +/// deleted neighbours each decide something (owner, 2026-10-11: a key reads every LIVE neighbour). +const LIVE_ORDERS: &str = r#" +name = "live-neighbours-fixture" +[description] +en = "x" +de = "y" +[priority] +labels = ["P0", "P1", "P2", "P3"] +[types] +list = ["goal", "project", "action"] +[vocabulary.status] +open = "open" +in_progress = "in progress" +closed = "closed" +[ranking.next] +types = ["project", "action"] +order = [ { field = "priority", dir = "asc" } ] +[ranking.next.project] +order = [ + { related = "contributes_to", type = "goal", of = { field = "order" }, pick = "min", nulls = "last" }, + { count = "children", dir = "desc" }, + { count = "mentioned_by", dir = "desc" }, +] +[ranking.next.action] +order = [ { related = "parent", type = "project", of = "rank", nulls = "last" } ] +[presentation.list] +columns = ["id"] +[fields.order] +type = "text" +on = ["goal"] +"#; + +#[test] +fn a_closed_archived_or_deleted_neighbour_reads_alike_with_and_without_a_store() { + // Goals: `gc` closed (order 1), `ga` archived (order 2), `go` open (order 3), `gd` deleted + // (order 0). Projects `pa`..`pd` contribute to one each: a closed or archived goal still gives + // its order, a deleted one gives none (null, last). Projects `q1`..`q4` have no goal: `q1` has + // three children — two closed, one archived — `q2` one open child, `q3` is mentioned by a + // closed project and `q4` by nothing. Every child and mentioner but `q2`'s is out of `next`. + let cfg: plugin::PluginConfig = toml::from_str(LIVE_ORDERS).unwrap(); + let mut s = Store::open_in_memory(1); + for (g, order) in [ + ("zz12.gc", "1"), + ("zz12.ga", "2"), + ("zz12.go", "3"), + ("zz12.gd", "0"), + ] { + item(&mut s, g, "goal", "1"); + custom(&mut s, g, "order", order); + } + close(&mut s, "zz12.gc"); + s.set_field("zz12.ga", "archived", Some("2026-06-01".into()), "t"); + for (p, g) in [ + ("zz12.pa", "zz12.gc"), + ("zz12.pb", "zz12.ga"), + ("zz12.pc", "zz12.gd"), + ("zz12.pd", "zz12.go"), + ] { + item(&mut s, p, "project", "1"); + edge(&mut s, p, g, EdgeKind::ContributesTo); + } + s.delete_item("zz12.gd", "t"); + for q in ["zz12.q1", "zz12.q2", "zz12.q3", "zz12.q4"] { + item(&mut s, q, "project", "1"); + } + for (a, parent) in [ + ("zz12.a1", "zz12.q1"), + ("zz12.a2", "zz12.q1"), + ("zz12.a3", "zz12.q1"), + ("zz12.a4", "zz12.q2"), + ] { + item(&mut s, a, "action", "1"); + s.set_parent(a, parent, "t").unwrap(); + } + close(&mut s, "zz12.a1"); + close(&mut s, "zz12.a2"); + s.set_field("zz12.a3", "archived", Some("2026-06-01".into()), "t"); + item(&mut s, "zz12.px", "project", "1"); + edge(&mut s, "zz12.px", "zz12.q3", EdgeKind::Mentions); + close(&mut s, "zz12.px"); + + let want = [ + "zz12.pa", "zz12.pb", "zz12.pd", "zz12.q1", "zz12.q2", "zz12.q3", "zz12.pc", "zz12.q4", + ]; + let store = read::next(&cfg, &s, NOW, None).unwrap(); + assert_eq!(of_type(&store, "project"), want, "the store path"); + let reads = read::next_ranking_reads(&cfg).unwrap().unwrap(); + let board = read::with_ranking(&s, read::active_board(&s).unwrap(), &reads).unwrap(); + let graph = board.ranking.as_ref().unwrap(); + for id in ["zz12.gc", "zz12.ga", "zz12.a1", "zz12.a3", "zz12.px"] { + assert!( + graph.neighbours.contains_key(id), + "{id} is a live neighbour" + ); + } + assert!( + !graph.neighbours.contains_key("zz12.gd"), + "a deleted one is not" + ); + assert!( + graph.edges.iter().all(|e| e.to != "zz12.gd"), + "nor an edge to it" + ); + assert_eq!( + ids(&read::next_active(&cfg, &board, NOW).unwrap()), + ids(&store) + ); +} + +#[test] +fn a_next_order_reading_the_rank_of_a_type_only_list_orders_is_refused_without_a_store() { + // Spec §12 point 1 lets a `next` key read the rank `[ranking.list]` gives a type. That rank is + // computed over the type's live items, which a board of active tickets does not hold: the + // store-free `next` refuses the plugin (owner, 2026-10-11), the store path ranks it. Any other + // derived `next` key ranks without a store. + let cfg: plugin::PluginConfig = toml::from_str( + r#" + name = "list-rank-fixture" + [description] + en = "x" + de = "y" + [priority] + labels = ["P0", "P1"] + [types] + list = ["action", "person"] + [vocabulary.status] + open = "open" + in_progress = "in progress" + closed = "closed" + [ranking.next] + order = [ { field = "priority" } ] + [ranking.next.action] + order = [ { related = "mentions", type = "person", of = "rank", nulls = "last" } ] + [ranking.list.person] + order = [ { field = "priority", dir = "asc" } ] + [presentation.list] + columns = ["id"] + "#, + ) + .unwrap(); + let mut s = Store::open_in_memory(1); + item(&mut s, "zz12.a1", "action", "1"); + item(&mut s, "zz12.u1", "person", "0"); + edge(&mut s, "zz12.a1", "zz12.u1", EdgeKind::Mentions); + let err = read::next_ranking_reads(&cfg).unwrap_err(); + assert_eq!(err.kind, ErrorKind::Validation); + assert!( + err.msg + .contains("[ranking.next.action] reads the rank of 'person'"), + "{}", + err.msg + ); + let board = read::active_board(&s).unwrap(); + assert_eq!( + read::next_active(&cfg, &board, NOW).unwrap_err().kind, + ErrorKind::Validation + ); + assert!( + read::next(&cfg, &s, NOW, None).is_ok(), + "the store path ranks it" + ); } // ---- each kind over hand-built boards (§9 "per kind") ------------------------------------- diff --git a/crates/fold-ddb/Cargo.toml b/crates/fold-ddb/Cargo.toml index 97c413c..27dc171 100644 --- a/crates/fold-ddb/Cargo.toml +++ b/crates/fold-ddb/Cargo.toml @@ -37,3 +37,6 @@ aws-sdk-dynamodb = { version = "1", optional = true } tokio = { version = "1", features = ["rt-multi-thread", "macros", "time"] } # The records differential test (6j6v.t1ym) parses a plugin fixture that declares a custom field. toml = "0.8" +# The derived-keys differential test (6j6v.1c5s) registers its plugin fixture, so `plugin::load` +# validates it as it validates a consumer's plugin. +inventory = "0.3" diff --git a/crates/fold-ddb/src/board.rs b/crates/fold-ddb/src/board.rs index 2ffc850..36aea0e 100644 --- a/crates/fold-ddb/src/board.rs +++ b/crates/fold-ddb/src/board.rs @@ -49,6 +49,19 @@ //! Only `contributes_to` adds get one: folding another edge's add costs nothing more, and folding //! an edge remove one `GetItem` of its add, to learn its kind. //! +//! # What an adjacency entry says +//! +//! Revision 3 (6j6v.1c5s) makes the `.adj##` entries self-describing, as the entries +//! under a ticket are: each copies the edge add's `from_id`, `to_id`, `kind`, `lamport` and `site` +//! and says whether the add is removed ([`REMOVED`]). Every edge add has one under each end, +//! whatever its kind and whatever the ends' activity, so ONE Query on a ticket's `.adj#` prefix +//! reads every edge touching it, in and out, with its presence — what the derived ranking keys read +//! ([`crate::records::with_ranking`]): inbound `contributes_to`, `mentions` both ways, the +//! `parent` edges a child's current parent is picked from. The entries keep their place and key; +//! only their attributes grow. Folding an edge add reads its remove (a `dep` or `parent` add read it +//! already, for its flag); folding an edge remove now reads the add and the remove and rewrites +//! both entries — two `GetItem`s and two writes, one when the edge is a loop. +//! //! # One writer per stream //! //! The rows a reducer describes converge whatever the order and however many folders write them. @@ -279,7 +292,7 @@ pub(crate) async fn after(table: &T, changes: &[Change]) -> Result<(), link_edge(table, tag).await?; } for tag in removed { - refresh_edge(table, tag).await?; + link_edge(table, tag).await?; } for (at, tag) in owned_tags { refresh_owned(table, &OWNED[at], tag).await?; @@ -304,7 +317,7 @@ pub(crate) async fn refresh_item(table: &T, id: &str) -> Result<(), T: for entry in table.query_prefix(&adjacency_prefix(id)).await? { if let Some(tag) = text(&entry, "tag") { if let Some(edge) = table.get(&tag_key("edge_adds", tag)).await? { - flag_edge(table, tag, edge, Some((id, active))).await?; + flag_edge(table, tag, edge, None, Some((id, active))).await?; } } } @@ -349,46 +362,68 @@ pub(crate) async fn refresh_item(table: &T, id: &str) -> Result<(), T: Ok(()) } -/// Record that edge `tag` touches its ends, then set its flag. -pub(crate) async fn link_edge(table: &T, tag: &str) -> Result<(), T::Error> { - let Some(edge) = table.get(&tag_key("edge_adds", tag)).await? else { - return Ok(()); - }; - let ends: BTreeSet<&str> = [text(&edge, "from_id"), text(&edge, "to_id")] +/// The cells of edge add `edge` an adjacency entry copies beside its tag and [`REMOVED`] (index +/// revision 3, 6j6v.1c5s): its ends, its kind and its coordinate. +const EDGE_COPIES: [&str; 5] = ["from_id", "to_id", "kind", "lamport", "site"]; + +/// The adjacency entries of edge add `tag` — one under each end, the same entry twice for an edge +/// from a ticket to itself — from the add's row and whether it is removed. +async fn write_adjacency( + table: &T, + tag: &str, + edge: &Row, + removed: bool, +) -> Result<(), T::Error> { + let mut set = vec![("tag".to_string(), Cell::Text(tag.to_string()))]; + for column in EDGE_COPIES { + match edge.get(column) { + None | Some(Cell::Null) => {} + Some(cell) => set.push((column.to_string(), cell.clone())), + } + } + set.push((REMOVED.to_string(), Cell::Int(i64::from(removed)))); + let ends: BTreeSet<&str> = [text(edge, "from_id"), text(edge, "to_id")] .into_iter() .flatten() .collect(); for end in ends { table - .write(&Write::put( - adjacency_key(end, tag), - vec![("tag".to_string(), Cell::Text(tag.to_string()))], - )) + .write(&Write::put(adjacency_key(end, tag), set.clone())) .await?; } - flag_edge(table, tag, edge, None).await + Ok(()) } -/// Set edge `tag`'s flag from its row, its remove and its ends. -pub(crate) async fn refresh_edge(table: &T, tag: &str) -> Result<(), T::Error> { - match table.get(&tag_key("edge_adds", tag)).await? { - Some(edge) => flag_edge(table, tag, edge, None).await, - None => Ok(()), - } +/// Record that edge `tag` touches its ends — the entry under each end says what the edge is and +/// whether it is removed — then set its flag. Run when the add is folded and again when its remove +/// is; nothing while the add has not been folded, since the add writes both. +pub(crate) async fn link_edge(table: &T, tag: &str) -> Result<(), T::Error> { + let Some(edge) = table.get(&tag_key("edge_adds", tag)).await? else { + return Ok(()); + }; + let removed = table.get(&tag_key("edge_removes", tag)).await?.is_some(); + write_adjacency(table, tag, &edge, removed).await?; + flag_edge(table, tag, edge, Some(removed), None).await } -/// `known` is a ticket whose activity the caller knows ahead of its stored flag. +/// `removed` is whether the edge's remove is folded, when the caller has read it already; `known` +/// a ticket whose activity the caller knows ahead of its stored flag. async fn flag_edge( table: &T, tag: &str, edge: Row, + removed: Option, known: Option<(&str, bool)>, ) -> Result<(), T::Error> { let indexed = matches!( text(&edge, "kind").and_then(EdgeKind::parse), Some(EdgeKind::Dep | EdgeKind::Parent) ); - let mut flag = indexed && table.get(&tag_key("edge_removes", tag)).await?.is_none(); + let mut flag = indexed + && match removed { + Some(removed) => !removed, + None => table.get(&tag_key("edge_removes", tag)).await?.is_none(), + }; if flag { let mut any_active = false; for end in [text(&edge, "from_id"), text(&edge, "to_id")] diff --git a/crates/fold-ddb/src/fold.rs b/crates/fold-ddb/src/fold.rs index 6d0c283..e4f8075 100644 --- a/crates/fold-ddb/src/fold.rs +++ b/crates/fold-ddb/src/fold.rs @@ -21,7 +21,9 @@ //! bumps it, and every stream folded before is no longer current and is folded again the same way. //! Revision 1 (6j6v.t1ym) adds the `.ladj#`/`.tadj#` entries; a stream folded before it has none, //! so its watermark names no index revision at all and the next run refolds it. Revision 2 -//! (6j6v.15ed) adds the `.cadj#` entries, a ticket's outgoing `contributes_to` edges. +//! (6j6v.15ed) adds the `.cadj#` entries, a ticket's outgoing `contributes_to` edges. Revision 3 +//! (6j6v.1c5s) makes the `.adj#` entries self-describing — ends, kind, coordinate, presence — for +//! the derived ranking keys ([`crate::records::with_ranking`]). //! //! # What the server cannot see //! @@ -46,7 +48,9 @@ pub const INDEX_DOMAIN: &str = "index"; /// The revision of those rules. 1: the entries under a ticket for its label and thread link adds /// (6j6v.t1ym). 2: the entries under a ticket for its outgoing `contributes_to` edges (6j6v.15ed). -pub const INDEX_REVISION: i64 = 2; +/// 3: the adjacency entries copy their edge's ends, kind and coordinate and say whether it is +/// removed (6j6v.1c5s). +pub const INDEX_REVISION: i64 = 3; /// The platform's reducers — the board's, the chat's and the memory's. pub fn platform_reducers() -> Vec> { diff --git a/crates/fold-ddb/src/lib.rs b/crates/fold-ddb/src/lib.rs index 868b37b..f97134f 100644 --- a/crates/fold-ddb/src/lib.rs +++ b/crates/fold-ddb/src/lib.rs @@ -23,6 +23,10 @@ //! | GSI `active` | `nxf_active` / `sk` | S/S | sparse: active tickets and the present dep/parent edges touching one | //! | GSI `dated` | `nxf_dated` / `nxf_dated_at` | S/S | sparse: `#closed` and `#archived`, by `␟` | //! +//! The `.adj##` entries are plain rows in the stream's partition, on no index. Since +//! index revision 3 (6j6v.1c5s) each also carries its edge's `from_id`, `to_id`, `kind`, `lamport`, +//! `site` and a `removed` flag — attributes, not keys: the contract above is unchanged. +//! //! Both indexes project ALL attributes, so a lane is one Query and no follow-up reads. The lanes //! need nothing else; the RECORDS a lane is shown with carry joins from other tables, which //! [`records`] reads with a fixed number of requests per active ticket. Billing mode @@ -61,7 +65,10 @@ //! one write. A label or thread link add, or its remove, adds two `GetItem`s and at most one write //! for the entry under the ticket (`board`, "Who owns a label or a thread link"). Reading the //! active tickets' records costs `4n + 1` requests for `n` tickets, plus one per parent outside -//! the selection, at most [`records::READ_CONCURRENCY`] tickets at a time ([`records`]). No request +//! the selection, at most [`records::READ_CONCURRENCY`] tickets at a time ([`records`]). Folding an +//! edge remove reads the add and the remove and rewrites the entry under each end. The graph the +//! derived ranking keys read ([`records::with_ranking`]) costs one Query per active ticket and, per +//! inactive neighbour, one `GetItem` plus up to three Queries. No request //! is a Scan: every read names its partition key, the board's through an index. The //! `dynamodb-local` CI job counts the requests of a fold and of a records read, and proves it. //! diff --git a/crates/fold-ddb/src/records.rs b/crates/fold-ddb/src/records.rs index cb930d8..4d52e05 100644 --- a/crates/fold-ddb/src/records.rs +++ b/crates/fold-ddb/src/records.rs @@ -34,6 +34,11 @@ //! a time, so a board's latency is about `n / READ_CONCURRENCY` round trips, and no board sends more //! than that many requests at once. //! +//! One more is read only for a plugin that ranks `next` by derived ranking keys: +//! [`with_ranking`] fills the board's [`RankingGraph`] — the edges around the active tickets and +//! their live neighbours — from the adjacency entries (index revision 3, 6j6v.1c5s). Its cost is in +//! its own doc. +//! //! # Two points in time //! //! The selection is an index read, eventually consistent; every join is a row read, strongly @@ -47,7 +52,8 @@ //! # A stream folded by older rules //! //! The entries under a ticket came with index revision 1, the `.cadj#` entries with revision 2 -//! (6j6v.15ed). A stream whose watermark names an older +//! (6j6v.15ed), the self-describing `.adj#` entries with revision 3 (6j6v.1c5s). A stream whose +//! watermark names an older //! one — or none, folded before revisions were recorded for the index — has no entries, and its //! records would come back without labels or conversations. [`active_board`] refuses such a stream //! ([`RecordsError::NotCurrent`]); the fold run refolds it ([`Folder::is_current`]), after which it @@ -58,12 +64,16 @@ use crate::board::{int, item_key, Selection, REMOVED}; use crate::fold::{Folder, INDEX_DOMAIN, INDEX_REVISION}; use crate::layout::{ - contributes_adjacency_prefix, label_adjacency_prefix, link_adjacency_prefix, row_key, + adjacency_prefix, contributes_adjacency_prefix, label_adjacency_prefix, link_adjacency_prefix, + row_key, }; use crate::table::{text, Row, Table}; use futures_util::stream::{self, StreamExt, TryStreamExt}; -use nexus_flow_core::model::{ItemRow, LinkWeight}; -use nexus_flow_facade::read::{ActiveBoard, ActiveTicket, ParentRef}; +use nexus_flow_core::graph::Edge; +use nexus_flow_core::model::{EdgeKind, ItemRow, LinkWeight}; +use nexus_flow_facade::read::{ + ActiveBoard, ActiveTicket, Neighbour, ParentRef, RankingGraph, RankingReads, +}; use nxs_foundation::change::Cell; use std::collections::{BTreeMap, BTreeSet}; @@ -206,6 +216,164 @@ async fn contributes_to(table: &T, id: &str) -> Result, T: Ok(targets.into_iter().collect()) } +/// One present edge as its adjacency entry tells it (index revision 3): the add's ends, kind and +/// coordinate (`-1` where the add has none, as `present_parent` reads it). +#[derive(Debug, Clone)] +struct EdgeEntry { + from: String, + to: String, + kind: EdgeKind, + lamport: i64, + site: i64, +} + +/// The present edges touching ticket `id`, by tag: one Query on its `.adj#` prefix. +async fn present_edges( + table: &T, + id: &str, +) -> Result, T::Error> { + Ok(present_entries(table, &adjacency_prefix(id)) + .await? + .iter() + .filter_map(|e| { + let entry = EdgeEntry { + from: text(e, "from_id")?.to_string(), + to: text(e, "to_id")?.to_string(), + kind: text(e, "kind").and_then(EdgeKind::parse)?, + lamport: int(e, "lamport").unwrap_or(-1), + site: int(e, "site").unwrap_or(-1), + }; + Some((text(e, "tag")?.to_string(), entry)) + }) + .collect()) +} + +/// The current parent of `id` among the present edges touching it — `present_parent`'s pick: of its +/// present outgoing `parent` edges, the one with the highest `(lamport, site, to_id)`. +fn current_parent(id: &str, edges: &BTreeMap) -> Option { + edges + .values() + .filter(|e| e.kind == EdgeKind::Parent && e.from == id) + .max_by(|a, b| (a.lamport, a.site, &a.to).cmp(&(b.lamport, b.site, &b.to))) + .map(|e| e.to.clone()) +} + +/// `board` with its [`RankingGraph`] for `reads` (6j6v.1c5s): the graph the derived ranking keys of +/// `[ranking.next]` read, from the folded tables — what [`read::with_ranking`] reads from a store, +/// edge for edge and neighbour for neighbour. A server calls it when +/// [`read::next_ranking_reads`] names reads for its plugin, so a plugin without derived `next` keys +/// pays nothing for it. The stream must be current, as for [`active_board`]: the entries read here +/// came with index revision 3. +/// +/// | read | per | +/// |-----------------------------------------------|-------------------------------------------| +/// | one Query on `.adj##` | active ticket | +/// | one `GetItem` of its row | other end of such an edge that is not active | +/// | one Query on `.adj##` (its parent) | such end that is live | +/// | one Query on `.ladj##`, `custom_fields##` | such end, when `reads` asks for labels, custom values | +/// +/// So `n` active tickets with `p` distinct inactive neighbours, `q` of them live, cost `n + q` +/// Queries and `p` `GetItem`s — `n + 3q` Queries with labels and custom values — at most +/// [`READ_CONCURRENCY`] items at a time, and nothing grows with an item's history: an entry says +/// whether its add is removed. Every edge touching an active ticket is read once, from one end's +/// entries, whatever the plugin's keys — the edge pass the store path runs once per query. +/// +/// [`read::with_ranking`]: nexus_flow_facade::read::with_ranking +/// [`read::next_ranking_reads`]: nexus_flow_facade::read::next_ranking_reads +pub async fn with_ranking( + table: &T, + mut board: ActiveBoard, + reads: &RankingReads, +) -> Result> { + // Every present edge of a kind read that touches an active ticket, once per tag. + let ids: Vec = board.tickets.keys().cloned().collect(); + let per_ticket: Vec> = stream::iter(ids) + .map(|id| async move { present_edges(table, &id).await }) + .buffered(READ_CONCURRENCY) + .try_collect() + .await?; + let mut touching: BTreeMap = BTreeMap::new(); + for edges in per_ticket { + touching.extend( + edges + .into_iter() + .filter(|(_, e)| reads.kinds.contains(&e.kind)), + ); + } + + // The other ends that are not active: their rows, and for the live ones their current parent + // and the joins asked for. + let others: BTreeSet = touching + .values() + .flat_map(|e| [&e.from, &e.to]) + .filter(|id| !board.tickets.contains_key(*id)) + .cloned() + .collect(); + let rows: Vec<(String, Option)> = stream::iter(others) + .map(|id| async move { + let row = table.get(&item_key(&id)).await?; + Ok::<_, T::Error>((id, row)) + }) + .buffered(READ_CONCURRENCY) + .try_collect() + .await?; + let live = rows + .into_iter() + .filter_map(|(id, row)| row.map(|r| (id, r))) + .filter(|(_, r)| text(r, "deleted") != Some("1")); + let neighbours: BTreeMap = stream::iter(live) + .map(|(id, row)| async move { + let parent = current_parent(&id, &present_edges(table, &id).await?); + let mut n = Neighbour::new(item_row(&row, parent)); + if reads.labels { + n.labels = labels(table, &id).await?; + } + if reads.custom { + n.custom = custom(table, &id).await?; + } + Ok::<_, T::Error>((id, n)) + }) + .buffered(READ_CONCURRENCY) + .try_collect() + .await?; + + // The edges between live ends, a `parent` edge only from a child to its current parent. + let parent_of = |id: &str| -> Option<&str> { + board + .tickets + .get(id) + .map(|t| &t.item) + .or_else(|| neighbours.get(id).map(|n| &n.item)) + .and_then(|i| i.belongs_to.as_deref()) + }; + let live_end = |id: &str| board.tickets.contains_key(id) || neighbours.contains_key(id); + let edges: BTreeSet = touching + .into_values() + .filter(|e| live_end(&e.from) && live_end(&e.to)) + .filter(|e| e.kind != EdgeKind::Parent || parent_of(&e.from) == Some(e.to.as_str())) + .map(|e| Edge { + from: e.from, + to: e.to, + kind: e.kind, + }) + .collect(); + // A child whose current parent is elsewhere was read for its parent edge only, which is not in + // the graph: it is no neighbour. + let ends: BTreeSet<&str> = edges + .iter() + .flat_map(|e| [e.from.as_str(), e.to.as_str()]) + .collect(); + let neighbours = neighbours + .into_iter() + .filter(|(id, _)| ends.contains(id.as_str())) + .collect(); + let mut graph = RankingGraph::new(reads.clone()); + graph.edges = edges; + graph.neighbours = neighbours; + board.ranking = Some(graph); + Ok(board) +} + /// An `items` row as the store reads it, with the ticket's `belongs_to` (the `present_parent` join). pub(crate) fn item_row(row: &Row, belongs_to: Option) -> ItemRow { let cell = |c: &str| text(row, c).map(str::to_string); diff --git a/crates/fold-ddb/tests/derived_differential.rs b/crates/fold-ddb/tests/derived_differential.rs new file mode 100644 index 0000000..cd5b228 --- /dev/null +++ b/crates/fold-ddb/tests/derived_differential.rs @@ -0,0 +1,483 @@ +//! A server ranks `next` by derived ranking keys as a replica does (6j6v.1c5s, +//! docs/specs/derived-ranking-keys.md §6 and §9 "differential"). +//! +//! Generated boards of three types — goals, projects, actions — written by two replicas apart and +//! then merged: every status, archived and deleted items, priorities, the goals' `order` custom +//! field, `waiting` labels, and `parent`, `contributes_to`, `mentions` and `dep` edges with removes. +//! Two replicas parent the same child apart, so a child can have two present parent edges and only +//! `present_parent`'s pick may count. Each board is folded into a [`MemTable`] in a shuffled order +//! with duplicates, as the relay hands it over. +//! +//! Then three paths rank `next` under a plugin whose orders hold every derived key kind, each +//! reading closed, archived and deleted neighbours, down a three-type chain (action → project → +//! goal): +//! +//! 1. the SQL path: `read::next` over the store; +//! 2. the Board path: `read::next_active` over the store's own [`read::ActiveBoard`] and +//! [`read::RankingGraph`] (`read::active_board` + `read::with_ranking`); +//! 3. the fold-ddb path: `read::next_active` over the folded tables' (`board::select` + +//! `records::active_board` + `records::with_ranking`). +//! +//! All three must give the same rows in the same order and the same JSON, and the two inputs must +//! be equal, edge for edge and neighbour for neighbour. Folding the same log in another order must +//! read the same graph (determinism). + +use nexus_flow_core::model::{EdgeKind, ItemRow, MergeStrategy}; +use nexus_flow_core::store::Store; +use nexus_flow_facade::plugin::{self, PluginConfig, PluginRegistration}; +use nexus_flow_facade::read; +use nxs_fold_ddb::fold::{Folder, INDEX_DOMAIN}; +use nxs_fold_ddb::layout::WATERMARK_KEY; +use nxs_fold_ddb::mem::{ready, MemTable}; +use nxs_fold_ddb::table::Table; +use nxs_fold_ddb::write::{Cond, Write}; +use nxs_fold_ddb::{board, records}; +use nxs_foundation::change::Cell; +use std::collections::{BTreeMap, BTreeSet}; + +mod common; +use common::{delivered, Lcg, NOW}; + +/// Every derived key kind, each able to read a neighbour that is not active: a goal's order and +/// rank from a closed or archived goal, counts without a status filter, a `dep` blocker's status +/// (only a closed blocker leaves its dependent in `next`), the `belongs_to` of a mentioned item. +const ORDERS: &str = r#" +name = "derived-differential-fixture" +[description] +en = "x" +de = "y" +[priority] +labels = ["P0", "P1", "P2", "P3", "P4"] +[types] +list = ["goal", "project", "action"] +[vocabulary.status] +open = "open" +in_progress = "in progress" +closed = "closed" +[ranking.next] +types = ["action", "project", "goal"] +order = [ { field = "priority", dir = "asc" } ] +[ranking.next.goal] +order = [ + { count = "contributors", type = "project", dir = "desc" }, + { field = "priority", dir = "asc" }, +] +[ranking.next.project] +order = [ + { related = "contributes_to", type = "goal", of = { field = "order" }, pick = "min", nulls = "last" }, + { related = "parent", type = "goal", of = "rank", pick = "max", nulls = "first" }, + { sum = [ { count = "mentioned_by" }, + { count = "mentioned_by", type = "action", label = "waiting", status = ["open", "in_progress"] } ], dir = "desc" }, + { count = "children", type = "action", dir = "desc" }, + { field = "priority", dir = "asc" }, +] +[ranking.next.action] +order = [ + { related = "parent", type = "project", of = "rank", nulls = "last" }, + { related = "dep", of = { field = "status", precedence = ["closed", "in_progress", "open"] }, pick = "min" }, + { count = "dependents", dir = "desc" }, + { related = "mentions", of = { field = "belongs_to" }, pick = "max", nulls = "last" }, + { count = "contributors", status = ["closed"], dir = "desc" }, + { field = "priority", dir = "asc" }, +] +[presentation.list] +columns = ["id"] +[fields.order] +type = "text" +on = ["goal"] +"#; + +inventory::submit!(PluginRegistration { + name: "derived-differential-fixture", + toml: ORDERS, + order: 140, +}); + +fn cfg() -> PluginConfig { + plugin::load("derived-differential-fixture").expect("the fixture is a valid plugin") +} + +/// The items of a generated board: three goals, five projects, eight actions. +fn items() -> Vec<(String, &'static str)> { + let mut out = Vec::new(); + for (ty, letter, n) in [("goal", 'g', 3), ("project", 'p', 5), ("action", 'a', 8)] { + for i in 0..n { + out.push((format!("ab12.{letter}{i:03}"), ty)); + } + } + out +} + +/// One replica's share of the writing. `added` holds the edges this replica added, so a remove +/// usually takes back a real one. +fn write(s: &mut Store, r: &mut Lcg, added: &mut Vec<(String, String, EdgeKind)>, steps: usize) { + let all = items(); + let pick = |r: &mut Lcg| all[r.next(all.len() as u64) as usize].0.clone(); + for step in 0..steps { + s.set_wall_clock(&format!( + "2026-10-0{}T{:02}:00:00Z", + 1 + r.next(7), + step % 24 + )); + let (x, y) = (pick(r), pick(r)); + match r.next(16) { + 0 => s.set_field(&x, "status", Some("in_progress".into()), "u"), + 1 | 2 => s.set_field(&x, "status", Some("closed".into()), "u"), + 3 => s.set_field(&x, "status", Some("open".into()), "u"), + 4 => s.set_field( + &x, + "archived", + Some(format!("2026-10-0{}", 1 + r.next(7))), + "u", + ), + 5 => s.set_field(&x, "archived", None, "u"), + 6 if r.next(3) == 0 => s.delete_item(&x, "u"), + 7 => s.set_field(&x, "priority", Some(r.next(5).to_string()), "u"), + 8..=12 => { + // Dependencies are rare: an active blocker takes its dependent out of `next`. + let kind = match r.next(10) { + 0..=2 => EdgeKind::Parent, + 3..=5 => EdgeKind::ContributesTo, + 6..=8 => EdgeKind::Mentions, + _ => EdgeKind::Dep, + }; + s.add_edge(&x, &y, kind, "u"); + added.push((x, y, kind)); + } + 13 if !added.is_empty() => { + let (from, to, kind) = added.swap_remove(r.next(added.len() as u64) as usize); + s.remove_edge(&from, &to, kind, "u"); + } + 14 => { + if r.next(3) == 0 { + s.remove_label(&x, "waiting", "u"); + } else { + s.add_label(&x, "waiting", "u"); + } + } + _ => s.set_custom_field_merge( + &x, + "order", + Some(r.next(6).to_string()), + "u", + MergeStrategy::Lww, + ), + } + } +} + +/// A merged board: two replicas that wrote apart, then exchanged everything. +fn gen_board(seed: u64) -> Store { + let mut r = Lcg(seed ^ 0xd1ff); + let mut a = Store::open_in_memory(1); + for (id, ty) in items() { + a.create_item(&id, ty, "T", "u"); + a.set_field(&id, "priority", Some(r.next(5).to_string()), "u"); + if ty == "goal" && r.next(4) != 0 { + a.set_custom_field_merge( + &id, + "order", + Some(r.next(6).to_string()), + "u", + MergeStrategy::Lww, + ); + } + } + let mut b = Store::open_in_memory(2); + b.apply(&a.export()); + let steps = 30 + r.next(40) as usize; + write(&mut a, &mut r, &mut Vec::new(), steps); + write(&mut b, &mut r, &mut Vec::new(), steps); + let from_b = b.export(); + a.apply(&from_b); + a +} + +/// `ops` folded into a fresh table. +fn fold(folder: &Folder, ops: &[(i64, nxs_foundation::model::Op)]) -> MemTable { + let t = MemTable::new("stream-1"); + ready(folder.fold_batch(&t, ops)).unwrap(); + t +} + +/// The served board with its ranking graph, as a server reads it for `reads` — and with the +/// contributes-to targets, so its tickets compare whole against `read::active_board`'s. +fn served(t: &MemTable, reads: &read::RankingReads) -> read::ActiveBoard { + let selection = ready(board::select(t)).unwrap(); + let plain = ready(records::active_board(t, selection)).unwrap(); + let plain = ready(records::with_contributes_to(t, plain)).unwrap(); + ready(records::with_ranking(t, plain, reads)).unwrap() +} + +fn ids(items: &[ItemRow]) -> Vec { + items.iter().map(|i| i.id.clone()).collect() +} + +fn count(s: &Store, sql: &str) -> usize { + s.connection() + .query_row(sql, [], |r| r.get::<_, i64>(0)) + .unwrap() as usize +} + +/// What the generated boards held, so the comparison is known to have compared something worth +/// comparing. +#[derive(Default)] +struct Seen(BTreeMap<&'static str, usize>); + +impl Seen { + fn add(&mut self, what: &'static str, n: usize) { + *self.0.entry(what).or_default() += n; + } + + fn assert_all(&self) { + for (what, n) in &self.0 { + assert!(*n > 0, "the generated boards have {what}: {:?}", self.0); + } + } +} + +#[test] +fn the_server_ranks_next_by_derived_keys_as_the_replica_does() { + let folder = Folder::platform().unwrap(); + let cfg = cfg(); + let reads = read::next_ranking_reads(&cfg) + .unwrap() + .expect("derived next keys"); + assert_eq!(reads.kinds.len(), 4, "every edge kind: {reads:?}"); + assert!(reads.labels && reads.custom, "{reads:?}"); + let mut seen = Seen::default(); + for seed in 0..60 { + let s = gen_board(seed); + let ctx = format!("seed {seed}"); + let t = fold(&folder, &delivered(&s, seed)); + + let local = read::with_ranking(&s, read::active_board(&s).unwrap(), &reads).unwrap(); + let server = served(&t, &reads); + assert_eq!(server.tickets, local.tickets, "active tickets, {ctx}"); + assert_eq!(server.ranking, local.ranking, "the ranking graph, {ctx}"); + + // 1 = 2 = 3: the SQL path, the Board path, the fold-ddb path. + let want = read::next(&cfg, &s, NOW, None).unwrap(); + assert_eq!( + ids(&read::next_active(&cfg, &local, NOW).unwrap()), + ids(&want), + "SQL path vs Board path, {ctx}" + ); + assert_eq!( + ids(&read::next_active(&cfg, &server, NOW).unwrap()), + ids(&want), + "SQL path vs fold-ddb path, {ctx}" + ); + assert_eq!( + read::next_active_value(&cfg, &server, NOW).unwrap(), + read::next_to_value_with_custom(&cfg, &s, &want).unwrap(), + "next JSON, {ctx}" + ); + + // Determinism: the same log folded in another order reads the same graph (every fourth + // board — a fold is most of a seed's time). + if seed % 4 == 0 { + let again = served(&fold(&folder, &delivered(&s, seed ^ 0xffff)), &reads); + assert_eq!(again.ranking, server.ranking, "another fold order, {ctx}"); + } + + // What the boards held. + let graph = server.ranking.as_ref().unwrap(); + let candidates: BTreeSet<&str> = want.iter().map(|i| i.id.as_str()).collect(); + seen.add("next rows", want.len()); + for n in graph.neighbours.values() { + let archived = n.item.archived.is_some(); + seen.add( + "closed neighbours", + usize::from(!archived && n.item.status.as_deref() == Some("closed")), + ); + seen.add("archived neighbours", usize::from(archived)); + seen.add("labelled neighbours", usize::from(!n.labels.is_empty())); + seen.add( + "goal neighbours with an order", + usize::from(n.custom.contains_key("order")), + ); + seen.add( + "neighbours with a parent", + usize::from(n.item.belongs_to.is_some()), + ); + } + for e in &graph.edges { + let into_candidate = candidates.contains(e.to.as_str()); + let from_neighbour = graph.neighbours.contains_key(&e.from); + seen.add( + "inbound contributes_to to a candidate", + usize::from(into_candidate && e.kind == EdgeKind::ContributesTo), + ); + seen.add( + "mentions of a candidate from an inactive item", + usize::from(into_candidate && from_neighbour && e.kind == EdgeKind::Mentions), + ); + seen.add( + "inactive children of a candidate", + usize::from(into_candidate && from_neighbour && e.kind == EdgeKind::Parent), + ); + seen.add( + "dep edges from a candidate to an inactive blocker", + usize::from( + candidates.contains(e.from.as_str()) + && graph.neighbours.contains_key(&e.to) + && e.kind == EdgeKind::Dep, + ), + ); + } + seen.add( + "edges dropped for a deleted end", + count( + &s, + "SELECT COUNT(*) FROM present_edges e JOIN items i + ON i.id IN (e.from_id, e.to_id) WHERE i.deleted = '1'", + ), + ); + // A child with two present parent edges: only `present_parent`'s pick may count. + seen.add( + "children parented twice", + count( + &s, + "SELECT COUNT(*) FROM (SELECT from_id FROM present_edges WHERE kind = 'parent' + GROUP BY from_id HAVING COUNT(*) > 1)", + ), + ); + seen.add( + "removed edges", + count(&s, "SELECT COUNT(*) FROM edge_removes"), + ); + // A derived key decides: a project ranked ahead of one with a better priority. + let projects: Vec<&ItemRow> = want + .iter() + .filter(|i| i.item_type.as_deref() == Some("project")) + .collect(); + seen.add( + "projects ordered against their priority", + projects + .windows(2) + .filter(|w| w[0].priority > w[1].priority) + .count(), + ); + } + seen.assert_all(); + for what in [ + "closed neighbours", + "archived neighbours", + "inbound contributes_to to a candidate", + "mentions of a candidate from an inactive item", + "inactive children of a candidate", + "dep edges from a candidate to an inactive blocker", + "edges dropped for a deleted end", + "children parented twice", + "projects ordered against their priority", + ] { + assert!(seen.0.contains_key(what), "{what} is counted"); + } +} + +#[test] +fn reading_the_ranking_graph_costs_one_query_per_ticket_and_per_live_neighbour() { + // §9 "bounded cost": every edge touching an active ticket is read once, from its entries; + // each inactive neighbour costs one GetItem and, when live, one Query for its parent plus one + // per join asked for — and nothing grows with history: an edge added and removed many times + // is still one entry per add, read in the same Query. + let folder = Folder::platform().unwrap(); + let cfg = cfg(); + let reads = read::next_ranking_reads(&cfg).unwrap().unwrap(); + let mut s = Store::open_in_memory(1); + s.create_item("ab12.g000", "goal", "Closed goal", "u"); + s.set_field("ab12.g000", "status", Some("closed".into()), "u"); + s.create_item("ab12.g001", "goal", "Deleted goal", "u"); + let n = 10u64; + for i in 0..n { + let id = format!("ab12.p{i:03}"); + s.create_item(&id, "project", "P", "u"); + for round in 0..5 { + s.add_edge(&id, "ab12.g000", EdgeKind::ContributesTo, "u"); + if round < 4 { + s.remove_edge(&id, "ab12.g000", EdgeKind::ContributesTo, "u"); + } + } + s.add_edge(&id, "ab12.g001", EdgeKind::Mentions, "u"); + } + s.delete_item("ab12.g001", "u"); + let t = fold(&folder, &delivered(&s, 1)); + let plain = ready(records::active_board(&t, ready(board::select(&t)).unwrap())).unwrap(); + let before = t.requests(); + let server = ready(records::with_ranking(&t, plain, &reads)).unwrap(); + let after = t.requests(); + // One adjacency Query per active ticket; the live neighbour's parent, labels and custom values. + assert_eq!(after.queries - before.queries, n + 3); + // One row per inactive neighbour: the closed goal and the deleted one. + assert_eq!(after.gets - before.gets, 2); + assert_eq!(after.index_queries, before.index_queries); + assert_eq!(after.writes, before.writes); + let local = read::with_ranking(&s, read::active_board(&s).unwrap(), &reads).unwrap(); + assert_eq!(server.ranking, local.ranking); + let graph = server.ranking.unwrap(); + assert_eq!( + graph.edges.len(), + n as usize, + "one contributes_to per project" + ); + assert_eq!( + graph.neighbours.keys().collect::>(), + ["ab12.g000"], + "the closed goal is a neighbour, the deleted one is not" + ); +} + +#[test] +fn a_stream_folded_by_index_revision_2_is_refused_then_refolded_and_ranks() { + // A stream folded before the adjacency entries said what their edge is (index revision 2): its + // entries hold only the tag, which would read no edge at all — so it is refused, and the fold + // run's rule refolds it. + let folder = Folder::platform().unwrap(); + let cfg = cfg(); + let reads = read::next_ranking_reads(&cfg).unwrap().unwrap(); + for seed in [3, 11, 27] { + let s = gen_board(seed); + let ops = delivered(&s, seed); + let t = fold(&folder, &ops); + for (sk, row) in t.rows() { + if let Some(Cell::Text(tag)) = row.get("tag").filter(|_| sk.starts_with(".adj#")) { + ready( + t.write(&Write { + sk: sk.clone(), + set: vec![("tag".into(), Cell::Text(tag.clone()))], + remove: ["from_id", "to_id", "kind", "lamport", "site", "removed"] + .map(String::from) + .to_vec(), + condition: Some(Cond::RowPresent), + }), + ) + .unwrap(); + } + } + ready(t.write(&Write { + sk: WATERMARK_KEY.into(), + set: vec![(format!("revision_{INDEX_DOMAIN}"), Cell::Int(2))], + remove: Vec::new(), + condition: Some(Cond::RowPresent), + })) + .unwrap(); + + let selection = ready(board::select(&t)).unwrap(); + match ready(records::active_board(&t, selection)) { + Err(records::RecordsError::NotCurrent { found: Some(2) }) => {} + other => panic!("seed {seed}: a revision-2 stream must be refused, got {other:?}"), + } + let mark = ready(Folder::watermark(&t)).unwrap().unwrap(); + assert!(!folder.is_current(&mark), "seed {seed}"); + ready(Folder::clear(&t)).unwrap(); + ready(folder.fold_batch(&t, &ops)).unwrap(); + + let server = served(&t, &reads); + assert_eq!( + ids(&read::next_active(&cfg, &server, NOW).unwrap()), + ids(&read::next(&cfg, &s, NOW, None).unwrap()), + "seed {seed}" + ); + } +} diff --git a/docs/specs/derived-ranking-keys.md b/docs/specs/derived-ranking-keys.md index 22c2d7f..cf37db7 100644 --- a/docs/specs/derived-ranking-keys.md +++ b/docs/specs/derived-ranking-keys.md @@ -2,7 +2,9 @@ **Status: decided by the owner on 2026-10-09** (§10). The cuts that build it are listed in §11. **Built locally (x0wf, §12):** the three key kinds, their load-time rules and the type chain over -the SQL path, in `next` and `list`. The server cuts (1c5s, xkr7) are open. +the SQL path, in `next` and `list`. **Built on the server for `next` (1c5s, §6):** the same keys +over the hosted board, with the rule the owner decided on 2026-10-11 (§10.5). The `typed` index and +a ranked hosted `list` (xkr7) are open. Fourth cut of `6j6v.kfs7` (cut with the owner on 2026-10-09). The three cuts before it add the plumbing this one ranks with: @@ -135,11 +137,41 @@ derived key pays nothing. `nxs-fold-ddb` answers `next` from a sparse index of ACTIVE tickets through `graph::Board` (6j6v.k7w7). Derived keys must give the same order there. -**`next`.** -- `Board` gains the edge kinds the keys need (`contributes_to`, `mentions`), and each ticket's - type, status, labels and the fields the keys read. -- The rule that makes this possible is the one `Board` already has: a counterpart that is not an - active ticket counts as absent, which is exactly §4's scope for `next`. +**`next`** (built in 1c5s; decided by the owner on 2026-10-11, §10.5). +- **A key reads every LIVE neighbour, on the server as locally** (§12, "Live neighbours"): a + deleted neighbour is gone, a closed or archived one counts. This replaces the first reading of + this section, that a counterpart which is not an active ticket counts as absent. That rule stays + the rule of the LANES (`graph::Board`); it is not the rule of the keys. Only the scope of + `of = "rank"` is the active set: a neighbour outside `next`'s candidates has no rank (§4), on + both paths. +- **Where it lives.** The facade's `ActiveBoard` — the store-free lanes' input, `graph::Board` + plus each active ticket's row and joins — gains a `RankingGraph`: every present edge of the kinds + the keys name (`contributes_to`, `mentions`, `dep`, `parent` as the `present_parent` projection) + with an active end and both ends live, and each live neighbour that is not active with its row, + labels and custom values. `graph::Board` itself is unchanged: it holds no rows, and the lanes read + no neighbour that is not active. `read::next_ranking_reads(cfg)` names what the graph must hold + for a plugin; `read::next_active` refuses a plugin with derived keys while its board lacks it, and + ranks by the same evaluation the store path runs once it is there. +- **What the fold writes.** Every edge add already had an adjacency entry under each end + (`.adj##`), whatever its kind and its ends' activity. Since index revision 3 each entry + also copies the add's `from_id`, `to_id`, `kind`, `lamport` and `site` and says whether the add is + removed, kept up to date when the remove is folded. So one Query on an item's prefix reads every + edge touching it, in and out, with its presence: inbound `contributes_to`, `mentions` both ways, + and the `parent` edges a child's current parent is picked from. No new index, no new key + attribute: the TABLE CONTRACT IS UNCHANGED. Revision 3 makes every stream folded before it not + current; it answers `NotCurrent` until the fold run has folded it again. +- **What the server reads** (`records::with_ranking`): one Query per active ticket on its + adjacency; per neighbour that is not active one `GetItem` of its row and, when it is live, one + Query on its own adjacency for its current parent, plus one Query each for its labels and custom + values when a key reads them. +- **Refused on the server until xkr7:** a `[ranking.next]` order that reads — directly or down its + type chain — the rank of a type only `[ranking.list]` orders (§12, first point). That rank is + computed over the type's live items, which the active board does not hold. The store path ranks + such a plugin. Every other derived `next` key ranks on the server. +- **What xkr7 builds on.** The self-describing adjacency entries are written for EVERY edge between + ANY items, so xkr7's "widened adjacency" needs no new entries for the edges: a `list`-ranked item + reads its edges with the same Query. What xkr7 adds is the `typed` index (the `list` scope of a + type), the read that ranks a type's live items from it, and lifting the refusal above. **`list` needs a second index (decided §10.4).** Objects that `list` ranks, such as a person, are long-lived and have no "active" status. If they did, they would rank through `next`, and `list` @@ -155,10 +187,12 @@ gains a second sparse index, keyed by type: - **Generic.** The fold indexes every live item by its type, so it needs no plugin knowledge of which types `list` ranks. The price is one index write per item, not only for the ranked types. - **Edges.** A derived key on a `list`-ranked type reads that item's `contributes_to` and `mentions` - neighbours, and the counts read their inbound edges. Today the adjacency entries (`.adj#`) are - kept only for `dep`/`parent` edges that touch an active ticket. They are widened to every present - edge of a kind some key names, between live items. That widening is the open design work of the - server cut (§11), together with what it costs per edge write. + neighbours, and the counts read their inbound edges. Corrected in 1c5s: the adjacency entries + (`.adj#`) were always written for every edge add, under both ends; only the `active` INDEX flag is + limited to `dep`/`parent` edges with an active end. Since index revision 3 the entries say what + their edge is and whether it is removed (`next` above), so a `list`-ranked item reads its edges + with one Query on its own prefix, at no extra cost per edge write. What stays open for xkr7 is + the read of a type's live items through `typed` and their neighbours' rows. - **Archived items.** They are in the index, because they are live. `list` filters them at read, as it does locally. - **It changes the table contract.** The deploying stack (the web package's CDK) must add the GSI @@ -222,8 +256,10 @@ the plugin declares (`plugin-custom-fields.md`), as decided in §10.2. - **Per kind:** unit tests over hand-built boards: a value from one neighbour, from several (`min`/`max`), from none (null placement); counts with each filter. - **Chains:** actions by project rank by goal order across three types, and a refused cycle. -- **Differential:** SQL path against `graph::Board` on generated boards with derived keys, as - `fold_differential` does for the lanes. +- **Differential:** SQL path against the server path on generated boards with derived keys, as + `fold_differential` does for the lanes. Built for `next` in 1c5s + (`crates/fold-ddb/tests/derived_differential.rs`): the SQL path, the store-free path over the + store's own board and graph, and the same over the folded tables give one order. - **Determinism:** shuffled edge order gives the same result, and ties fall to the id. - **Bounded cost:** a ratio guard in the style of `xn8s`. A query with derived keys runs the edge pass once, not once per candidate. @@ -245,13 +281,24 @@ the plugin declares (`plugin-custom-fields.md`), as decided in §10.2. items was declined: the objects `list` ranks have no active status by design. If they had one, they would rank through `next`, and `list` would not be needed. Leaving `list` unranked on the server was declined because the product reads through the hosted path. +5. **On the server a key reads every live neighbour** (owner, 2026-10-11, during 1c5s). The first + text of §6 said a counterpart that is not an active ticket counts as absent; the local build + (§12) reads every live neighbour, closed and archived ones included. The owner chose the local + rule for both paths, so a server and a replica rank alike: the fold keeps self-describing + adjacency entries (index revision 3) and the server reads the neighbours that are not active. + Narrowing the local rule to active neighbours was declined, as was refusing on the server every + key whose value can depend on a neighbour that is not active (it would refuse §8's first order). + A `[ranking.next]` order that reads the rank of a type only `[ranking.list]` orders is refused on + the server until the `typed` index (xkr7) gives it that type's scope. ## 11. Cuts - **Derived keys, local.** The three key kinds, load-time validation and the type chain over the SQL path. Usable in `next` (after `z9jk`) and `list` (after `n698`). -- **Derived keys on the server, `next`.** `graph::Board` with the extra edges and fields; the - differential test. +- **Derived keys on the server, `next`** (1c5s, built). The `RankingGraph` on the store-free + lanes' board, the self-describing adjacency entries (index revision 3), and the differential test + (SQL path = Board path = fold-ddb path on generated boards with closed, archived and deleted + neighbours, mentions and inbound `contributes_to`). - **The `typed` index and a ranked hosted `list`.** The GSI, the widened adjacency, the read, and the table-contract note for the deploying stack, coordinated with app-foundations. - **Shared with `6j6v.q29p`.** The reverse mention lookup and `count = "mentioned_by"` use one @@ -286,7 +333,8 @@ reading of the text, so a later decision can loosen it without breaking a declar `blocked` and the lanes' `--sort rank`, which compute none. The `[ranking.list]` default order takes them. - **`nulls` on a `count` or `sum`** is refused: a count is never null (§3.2). -- **The store-free lanes** (`read::next_active`) refuse a `[ranking.next]` with derived keys until - the server cut (1c5s) gives them the edges. +- **The store-free lanes** (`read::next_active`) rank a `[ranking.next]` with derived keys from + the board's `RankingGraph` since 1c5s (§6), with the live-neighbour rule above; they refuse only + an order that reads the rank of a type only `[ranking.list]` orders, until xkr7. - **TOML.** An inline table may not span lines outside its arrays (TOML 1.0), so a `sum`'s `dir` stands on the line its array closes on. §8 above breaks that line for reading only. From 9fb48191d0dd9900f0cd79d46bdd863a46f96902 Mon Sep 17 00:00:00 2001 From: Carsten Koch Date: Sun, 11 Oct 2026 20:04:29 +0200 Subject: [PATCH 2/3] The typed index and a ranked hosted list (nxf 6j6v.xkr7) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit A hosted board answers `list` ranked by `[ranking.list]`, derived keys included, as a replica does, and a hosted `next` whose keys read a `[ranking.list]` rank no longer is refused (spec §6, decision §10.4). - fold-ddb: third sparse GSI `typed` (nxf_typed = #, the type escaped; sort key sk; projection ALL) on every live item row, closed and archived included, kept by refresh_item after the row. Table::query_typed (MemTable, DynamoDbTable, create_table). A type whose partition passes 2,048 bytes is refused at fold time. Ships in index revision 3 with the self-describing .adj# entries (#54, unreleased): no revision 4. records::list_board reads a ListBoard; the graph builder is shared with with_ranking and reads the live items of RankingReads::list_types off the index. - facade: ListReads, ListItem, ListBoard, list_reads, list_board (store oracle), list_board_page, list_board_value; RankingReads::list_types, RankingGraph::listed (additive); next_ranking_reads no longer refuses a next order reading a list-only type's rank. Not paged, as the hosted next: limit cuts, paginate/token are validation. - No new adjacency: the revision-3 .adj# entries carry everything. - tests: list_differential (SQL = Board = fold-ddb for list and next on generated boards with retypes, cross-section ranks, JSON, cost guard, revision refusal), typed index in fold_differential, robustness and dynamodb_local, facade seam tests. - docs: crate table contract, E4 §5.1, spec §6/§9/§11/§12; 1c5s fragment aligned (contract change and lifted refusal). Co-Authored-By: Claude Opus 5.5 Claude-Session: https://claude.ai/code/session_016q5uMUkZJQW4fckJoA3FME --- changes/1c5s-server-derived-keys.md | 4 +- changes/xkr7-typed-index.md | 8 + crates/facade/src/read.rs | 380 ++++++++++++++++-- crates/facade/src/read/derived.rs | 402 ++++++++++++++----- crates/facade/tests/derived_ranking.rs | 196 ++++++++- crates/fold-ddb/src/board.rs | 49 ++- crates/fold-ddb/src/ddb.rs | 27 +- crates/fold-ddb/src/fold.rs | 23 +- crates/fold-ddb/src/layout.rs | 17 +- crates/fold-ddb/src/lib.rs | 36 +- crates/fold-ddb/src/mem.rs | 16 +- crates/fold-ddb/src/records.rs | 409 ++++++++++++++++--- crates/fold-ddb/src/table.rs | 11 +- crates/fold-ddb/tests/common/mod.rs | 11 +- crates/fold-ddb/tests/dynamodb_local.rs | 20 +- crates/fold-ddb/tests/fold_differential.rs | 28 +- crates/fold-ddb/tests/list_differential.rs | 441 +++++++++++++++++++++ crates/fold-ddb/tests/robustness.rs | 108 ++++- docs/specs/E4-sync-protocol-server.md | 7 + docs/specs/derived-ranking-keys.md | 74 +++- 20 files changed, 2020 insertions(+), 247 deletions(-) create mode 100644 changes/xkr7-typed-index.md create mode 100644 crates/fold-ddb/tests/list_differential.rs diff --git a/changes/1c5s-server-derived-keys.md b/changes/1c5s-server-derived-keys.md index 1d63ec8..58aed9e 100644 --- a/changes/1c5s-server-derived-keys.md +++ b/changes/1c5s-server-derived-keys.md @@ -3,6 +3,6 @@ type: added facade: changed --- [en] -A hosted board now ranks `next` by derived ranking keys (`related`, `count`, `sum`) exactly as a local replica does. The store-free `next` (`read::next_active`, `next_active_value`, `next_active_filtered`) ranks a plugin with such keys from a new `ActiveBoard::ranking` graph: the edges around the active tickets and the live neighbours they reach, closed and archived ones included, deleted ones not. A server fills it with `nxs_fold_ddb::records::with_ranking` for what `read::next_ranking_reads(cfg)` names; `read::with_ranking` reads the same graph from a store. Without the graph, `next_active` still refuses such a plugin. One case stays refused on the store-free path: a `[ranking.next]` order that reads the rank of a type only `[ranking.list]` orders — answer that `next` from a store. Everything is additive: `ActiveBoard` gains a field (it is `#[non_exhaustive]`), and `RankingReads`, `RankingGraph`, `Neighbour`, `next_ranking_reads` and `with_ranking` are new. For a server's operators: the fold's index revision goes from 2 to 3, so every hosted stream is not current and is folded again on its next run; until then its records read answers `NotCurrent`. The table contract is unchanged: no new index and no new key attribute, only more attributes on the existing `.adj#` adjacency entries. +A hosted board now ranks `next` by derived ranking keys (`related`, `count`, `sum`) exactly as a local replica does. The store-free `next` (`read::next_active`, `next_active_value`, `next_active_filtered`) ranks a plugin with such keys from a new `ActiveBoard::ranking` graph: the edges around the active tickets and the live neighbours they reach, closed and archived ones included, deleted ones not. A server fills it with `nxs_fold_ddb::records::with_ranking` for what `read::next_ranking_reads(cfg)` names; `read::with_ranking` reads the same graph from a store. Without the graph, `next_active` still refuses such a plugin. Everything is additive: `ActiveBoard` gains a field (it is `#[non_exhaustive]`), and `RankingReads`, `RankingGraph`, `Neighbour`, `next_ranking_reads` and `with_ranking` are new. For a server's operators: the fold's index revision goes from 2 to 3, so every hosted stream is not current and is folded again on its next run; until then its records read answers `NotCurrent`. The `.adj#` adjacency entries gain attributes; the one table-contract change of this release, the `typed` index, is described with the hosted `list`. [de] -Ein gehostetes Brett rankt `next` jetzt nach abgeleiteten Ranking-Schlüsseln (`related`, `count`, `sum`) genau so wie ein lokales Replikat. Das speicherfreie `next` (`read::next_active`, `next_active_value`, `next_active_filtered`) rankt ein Plugin mit solchen Schlüsseln aus einem neuen Graphen `ActiveBoard::ranking`: den Kanten um die aktiven Tickets und den lebenden Nachbarn, die sie erreichen, geschlossene und archivierte eingeschlossen, gelöschte nicht. Ein Server füllt ihn mit `nxs_fold_ddb::records::with_ranking` für das, was `read::next_ranking_reads(cfg)` nennt; `read::with_ranking` liest denselben Graphen aus einem Speicher. Ohne den Graphen lehnt `next_active` ein solches Plugin weiterhin ab. Ein Fall bleibt auf dem speicherfreien Weg abgelehnt: eine Reihenfolge in `[ranking.next]`, die den Rang eines Typs liest, den nur `[ranking.list]` ordnet — dieses `next` beantwortet ein Speicher. Alles ist additiv: `ActiveBoard` bekommt ein Feld (es ist `#[non_exhaustive]`), und `RankingReads`, `RankingGraph`, `Neighbour`, `next_ranking_reads` und `with_ranking` sind neu. Für den Betrieb eines Servers: Die Index-Revision des Folds steigt von 2 auf 3, daher ist jeder gehostete Stream nicht aktuell und wird bei seinem nächsten Lauf neu gefaltet; bis dahin antwortet sein Records-Lesen mit `NotCurrent`. Der Tabellenvertrag bleibt unverändert: kein neuer Index und kein neues Schlüsselattribut, nur mehr Attribute an den bestehenden `.adj#`-Adjazenzeinträgen. +Ein gehostetes Brett rankt `next` jetzt nach abgeleiteten Ranking-Schlüsseln (`related`, `count`, `sum`) genau so wie ein lokales Replikat. Das speicherfreie `next` (`read::next_active`, `next_active_value`, `next_active_filtered`) rankt ein Plugin mit solchen Schlüsseln aus einem neuen Graphen `ActiveBoard::ranking`: den Kanten um die aktiven Tickets und den lebenden Nachbarn, die sie erreichen, geschlossene und archivierte eingeschlossen, gelöschte nicht. Ein Server füllt ihn mit `nxs_fold_ddb::records::with_ranking` für das, was `read::next_ranking_reads(cfg)` nennt; `read::with_ranking` liest denselben Graphen aus einem Speicher. Ohne den Graphen lehnt `next_active` ein solches Plugin weiterhin ab. Alles ist additiv: `ActiveBoard` bekommt ein Feld (es ist `#[non_exhaustive]`), und `RankingReads`, `RankingGraph`, `Neighbour`, `next_ranking_reads` und `with_ranking` sind neu. Für den Betrieb eines Servers: Die Index-Revision des Folds steigt von 2 auf 3, daher ist jeder gehostete Stream nicht aktuell und wird bei seinem nächsten Lauf neu gefaltet; bis dahin antwortet sein Records-Lesen mit `NotCurrent`. Die `.adj#`-Adjazenzeinträge bekommen Attribute; die eine Änderung des Tabellenvertrags in diesem Release, der Index `typed`, ist bei der gehosteten `list` beschrieben. diff --git a/changes/xkr7-typed-index.md b/changes/xkr7-typed-index.md new file mode 100644 index 0000000..31fb4a1 --- /dev/null +++ b/changes/xkr7-typed-index.md @@ -0,0 +1,8 @@ +--- +type: added +facade: changed +--- +[en] +A hosted board now answers `list` ranked by the plugin's `[ranking.list]`, derived ranking keys included, exactly as a local replica does. The facade names what to read for a query (`read::list_reads`, a new `ListReads`), a server reads it into a new `ListBoard` with `nxs_fold_ddb::records::list_board`, and `read::list_board_page` / `read::list_board_value` filter (type, container, label, status), rank and project it with the code `list` runs locally; `read::list_board` reads the same board from a store. The hosted list is not paged: `limit` cuts it with the total disclosed, and `paginate` or a token is refused, as for the hosted `next`. A hosted `next` whose keys read the rank `[ranking.list]` gives a type now ranks too instead of being refused: `RankingReads` gains `list_types` and `RankingGraph` gains `listed` (both `#[non_exhaustive]`, so additive). For a server's operators, the table contract changes: a third sparse index, GSI `typed` — partition key `nxf_typed` (string, `#`), sort key `sk` (string), projection ALL — set by the fold on every live item row (closed and archived included, removed on delete). Add the index to the stack first, then deploy the read; a fold against a table without the index writes the attribute harmlessly. It ships in fold index revision 3 together with the self-describing adjacency entries, so one refold covers both. +[de] +Ein gehostetes Brett beantwortet `list` jetzt nach dem `[ranking.list]` des Plugins gerankt, abgeleitete Ranking-Schlüssel eingeschlossen, genau so wie ein lokales Replikat. Die Fassade nennt, was für eine Anfrage zu lesen ist (`read::list_reads`, ein neues `ListReads`), ein Server liest es mit `nxs_fold_ddb::records::list_board` in ein neues `ListBoard`, und `read::list_board_page` / `read::list_board_value` filtern (Typ, Container, Label, Status), ranken und projizieren es mit dem Code, den `list` lokal ausführt; `read::list_board` liest dasselbe Brett aus einem Speicher. Die gehostete Liste wird nicht geblättert: `limit` schneidet sie mit offengelegter Gesamtzahl, `paginate` oder ein Token wird abgelehnt, wie beim gehosteten `next`. Ein gehostetes `next`, dessen Schlüssel den Rang lesen, den `[ranking.list]` einem Typ gibt, rankt jetzt ebenfalls, statt abgelehnt zu werden: `RankingReads` bekommt `list_types` und `RankingGraph` bekommt `listed` (beide `#[non_exhaustive]`, also additiv). Für den Betrieb eines Servers ändert sich der Tabellenvertrag: ein dritter dünn besetzter Index, GSI `typed` — Partitionsschlüssel `nxf_typed` (String, `#`), Sortierschlüssel `sk` (String), Projektion ALL —, den der Fold an jeder lebenden Item-Zeile setzt (geschlossene und archivierte eingeschlossen, beim Löschen entfernt). Zuerst den Index in den Stack aufnehmen, dann das Lesen ausrollen; ein Fold gegen eine Tabelle ohne den Index schreibt das Attribut folgenlos. Er kommt mit der Fold-Index-Revision 3 zusammen mit den selbstbeschreibenden Adjazenzeinträgen, ein Neufalten deckt also beides ab. diff --git a/crates/facade/src/read.rs b/crates/facade/src/read.rs index 7784967..6f3ec09 100644 --- a/crates/facade/src/read.rs +++ b/crates/facade/src/read.rs @@ -2102,6 +2102,20 @@ fn blocked_record(cfg: Option<&PluginConfig>, row: &BlockedItem, j: &RecordJoins rec } +/// One `list` record without a store (6j6v.xkr7): what [`list_to_value_with_custom`] gives — the +/// canonical item, the sparse `labels` and `parent_closed_reason` joins, and the lane joins. +fn list_record(cfg: Option<&PluginConfig>, item: &ItemRow, j: &RecordJoins) -> Value { + let mut rec = item_value(item); + if !j.labels.is_empty() { + rec["labels"] = json!(j.labels); + } + if let Some(reasons) = parent_closed_reason_value(j.parent_closed) { + rec["parent_closed_reason"] = reasons; + } + put_lane_joins(cfg, &mut rec, j); + rec +} + /// One record of a bare lane (`deferred`/`closed`/`archived`/`search`): the canonical item and the /// lane joins. fn lane_record(cfg: Option<&PluginConfig>, item: &ItemRow, j: &RecordJoins) -> Value { @@ -2800,6 +2814,10 @@ pub struct RankingReads { pub labels: bool, /// An `of` reads a custom field: the neighbours' custom values. pub custom: bool, + /// The types whose `[ranking.list]` rank a key reads (6j6v.xkr7): the graph also holds every + /// live, unarchived item of each — the type's `list` scope ([`RankingGraph::listed`]) — and the + /// edges around them. A server reads them from its `typed` index. + pub list_types: BTreeSet, } impl RankingReads { @@ -2808,21 +2826,29 @@ impl RankingReads { needed.kinds.is_subset(&self.kinds) && (self.labels || !needed.labels) && (self.custom || !needed.custom) + && needed.list_types.is_subset(&self.list_types) } } -/// The graph around a board's active tickets that the derived ranking keys read (6j6v.1c5s, +/// The graph around a board's rows that the derived ranking keys read (6j6v.1c5s, /// docs/specs/derived-ranking-keys.md §6): the input [`next_active`] ranks a `[ranking.next]` with -/// `related` / `count` / `sum` keys from, so a server ranks exactly as [`next`] does over a store. +/// `related` / `count` / `sum` keys from, and [`list_board_page`] a `[ranking.list]` (6j6v.xkr7), +/// so a server ranks exactly as [`next`] and [`list_page`] do over a store. +/// +/// The SUBJECTS are the rows a key is evaluated on: the board's own rows — an [`ActiveBoard`]'s +/// active tickets; a [`ListBoard`]'s listed items and, when it carries one, its active board's +/// tickets — and the `listed` items below. /// -/// - `edges`: every present edge of a kind in `reads.kinds` with at least one ACTIVE end (an -/// [`ActiveBoard::tickets`] entry) and both ends LIVE (not deleted, not missing). A `parent` edge -/// is the `present_parent` projection — a child's one current parent, the one `belongs_to` reads — -/// so a child re-parented concurrently is under one parent only. Distinct, ordered. -/// - `neighbours`: the other ends of `edges` that are not active tickets — closed and archived -/// items included, since a key reads every LIVE neighbour (owner, 2026-10-11) — each with its row -/// (`belongs_to` filled as the store fills it), its labels when `reads.labels` and its non-empty -/// custom values when `reads.custom`, else empty. +/// - `edges`: every present edge of a kind in `reads.kinds` with at least one end a subject and +/// both ends LIVE (not deleted, not missing). A `parent` edge is the `present_parent` projection — +/// a child's one current parent, the one `belongs_to` reads — so a child re-parented concurrently +/// is under one parent only. Distinct, ordered. +/// - `listed`: the ids of every live, unarchived item of a type in `reads.list_types` — the `list` +/// scope a `[ranking.list]` rank is computed over (6j6v.xkr7). Empty without list types. +/// - `neighbours`: every live row the board does not hold itself — the other ends of `edges` and +/// the `listed` items, closed and archived items included, since a key reads every LIVE neighbour +/// (owner, 2026-10-11) — each with its row (`belongs_to` filled as the store fills it), its +/// labels when `reads.labels` and its non-empty custom values when `reads.custom`, else empty. /// /// `#[non_exhaustive]`: build it with [`RankingGraph::new`] and set the fields. #[derive(Debug, Clone, Default, PartialEq, Eq)] @@ -2833,6 +2859,8 @@ pub struct RankingGraph { pub edges: BTreeSet, /// By id. pub neighbours: BTreeMap, + /// The live, unarchived items of `reads.list_types` (6j6v.xkr7), by id. + pub listed: BTreeSet, } impl RankingGraph { @@ -2845,8 +2873,8 @@ impl RankingGraph { } } -/// A live item next to an active ticket that is not itself active: what a derived ranking key reads -/// of it ([`RankingGraph::neighbours`]). +/// A live item a derived ranking key reads that the board does not hold itself — next to one of +/// its rows, or in the `list` scope of a type ([`RankingGraph::neighbours`]). /// /// `#[non_exhaustive]`: build it with [`Neighbour::new`] and set the fields. #[derive(Debug, Clone, PartialEq, Eq)] @@ -2874,10 +2902,10 @@ impl Neighbour { /// [`RankingReads`] a reader fills [`ActiveBoard::ranking`] for — or `None` when `[ranking.next]` /// declares no derived ranking key (6j6v.1c5s). It does not depend on the board. /// -/// `validation` for a `[ranking.next]` that reads, directly or down its type chain, the rank of a -/// type only `[ranking.list]` orders (docs/specs/derived-ranking-keys.md §6): that rank is computed -/// over the type's live items, and a board of active tickets does not hold them. The store path -/// ([`next`]) ranks such a plugin. +/// A `[ranking.next]` that reads, directly or down its type chain, the rank of a type only +/// `[ranking.list]` orders names that type in [`RankingReads::list_types`] (6j6v.xkr7): that rank +/// is computed over the type's live items, which the reader puts in the graph. Until 6j6v.xkr7 +/// such a plugin was refused here; it is no longer, and the `Result` stays for a later refusal. pub fn next_ranking_reads(cfg: &PluginConfig) -> Result> { derived::reads_for_next(cfg) } @@ -2885,23 +2913,49 @@ pub fn next_ranking_reads(cfg: &PluginConfig) -> Result> { /// `board` with its [`RankingGraph`] for `reads` read from `store` (6j6v.1c5s) — the parity oracle /// for `nxs_fold_ddb::records::with_ranking`, as [`active_board`] is for the records. One edge pass /// ([`Store::present_edges_of_kinds`]), one bulk read of the neighbours' rows, and one of their -/// labels and of their custom values when `reads` asks for them. +/// labels and of their custom values when `reads` asks for them; with list types, one read of the +/// live items (6j6v.xkr7). pub fn with_ranking( store: &Store, mut board: ActiveBoard, reads: &RankingReads, ) -> Result { + let held: BTreeSet<&str> = board.tickets.keys().map(String::as_str).collect(); + board.ranking = Some(ranking_graph(store, reads, &held)?); + Ok(board) +} + +/// The [`RankingGraph`] for `reads` around the rows a board holds itself (`held`, every one a +/// subject), read from `store` — what [`with_ranking`] and [`list_board`] put on their boards. +fn ranking_graph( + store: &Store, + reads: &RankingReads, + held: &BTreeSet<&str>, +) -> Result { + let listed: Vec = if reads.list_types.is_empty() { + Vec::new() + } else { + derived::list_scope(store)? + .into_iter() + .filter(|i| { + i.item_type + .as_deref() + .is_some_and(|t| reads.list_types.contains(t)) + }) + .collect() + }; + let listed_ids: BTreeSet<&str> = listed.iter().map(|i| i.id.as_str()).collect(); + let subject = |id: &str| held.contains(id) || listed_ids.contains(id); let kinds: Vec = reads.kinds.iter().copied().collect(); - let active = |id: &str| board.tickets.contains_key(id); let touching: Vec<(String, String, EdgeKind)> = store .present_edges_of_kinds(&kinds)? .into_iter() - .filter(|(from, to, _)| active(from) || active(to)) + .filter(|(from, to, _)| subject(from) || subject(to)) .collect(); let others: BTreeSet<&str> = touching .iter() .flat_map(|(from, to, _)| [from.as_str(), to.as_str()]) - .filter(|id| !active(id)) + .filter(|id| !subject(id)) .collect(); let others: Vec<&str> = others.into_iter().collect(); let live: BTreeMap = store @@ -2913,11 +2967,17 @@ pub fn with_ranking( let edges: BTreeSet = touching .into_iter() .filter(|(from, to, _)| { - (active(from) || live.contains_key(from)) && (active(to) || live.contains_key(to)) + (subject(from) || live.contains_key(from)) && (subject(to) || live.contains_key(to)) }) .map(|(from, to, kind)| nexus_flow_core::graph::Edge { from, to, kind }) .collect(); - let ids: Vec<&str> = live.keys().map(String::as_str).collect(); + let mut rows = live; + for item in &listed { + if !held.contains(item.id.as_str()) { + rows.insert(item.id.clone(), item.clone()); + } + } + let ids: Vec<&str> = rows.keys().map(String::as_str).collect(); let mut labels = if reads.labels { store.labels_of_bulk(&ids)? } else { @@ -2930,7 +2990,8 @@ pub fn with_ranking( }; let mut graph = RankingGraph::new(reads.clone()); graph.edges = edges; - graph.neighbours = live + graph.listed = listed_ids.iter().map(|id| id.to_string()).collect(); + graph.neighbours = rows .into_iter() .map(|(id, item)| { let mut n = Neighbour::new(item); @@ -2939,8 +3000,7 @@ pub fn with_ranking( (id, n) }) .collect(); - board.ranking = Some(graph); - Ok(board) + Ok(graph) } /// The store path's input as an [`ActiveBoard`]: the store's selection ([`graph::select`]) and, for @@ -2987,10 +3047,11 @@ pub fn active_board(store: &Store) -> Result { /// [`Engine::next`]: crate::engine::Engine::next /// /// A `[ranking.next]` with derived ranking keys (6j6v.x0wf) is ranked from the board's -/// [`RankingGraph`] (6j6v.1c5s), by the same evaluation the store path runs. It is refused with -/// `validation` when the board carries no graph, or one read for less than -/// [`next_ranking_reads`] names — ranking without those edges would order apart from the store path -/// in silence — and when [`next_ranking_reads`] refuses the plugin. +/// [`RankingGraph`] (6j6v.1c5s), by the same evaluation the store path runs — a key that reads the +/// rank of a type only `[ranking.list]` orders too, from the graph's listed items (6j6v.xkr7). It +/// is refused with `validation` when the board carries no graph, or one read for less than +/// [`next_ranking_reads`] names: ranking without those edges would order apart from the store path +/// in silence. pub fn next_active(cfg: &PluginConfig, board: &ActiveBoard, now: &str) -> Result> { crate::validate::iso_date(now)?; let graph = match next_ranking_reads(cfg)? { @@ -3115,6 +3176,267 @@ pub fn deferred_active_value(cfg: &PluginConfig, board: &ActiveBoard, now: &str) Ok(board.records(&items, |i, j| lane_record(Some(cfg), i, j))) } +// ---- list without a store (6j6v.xkr7) ---------------------------------------- +// +// The hosted `list`: the same rows in the same order as [`list_page`] over a store holding the same +// board, from a [`ListBoard`] a server reads off its `typed` index (docs/specs/derived-ranking-keys.md +// §6) — the listed types' live items with their joins, `next`'s board when a key reads a rank +// `next` gives, and the [`RankingGraph`] the derived keys read. As for the store-free `next`, the +// facade decides what to read ([`list_reads`]) and ranks and projects it; the server only reads. + +/// What the store-free `list` a [`ListQuery`] asks for reads (6j6v.xkr7) — the input of a reader +/// that builds the [`ListBoard`] ([`list_board`] from a store, `nxs_fold_ddb::records::list_board` +/// from the folded tables). [`list_reads`] gives it. +/// +/// `#[non_exhaustive]`: start from `ListReads::default()` and set the fields. +#[derive(Debug, Clone, Default, PartialEq, Eq)] +#[non_exhaustive] +pub struct ListReads { + /// The listed types: the board holds every live, unarchived item of each. `None`: every live, + /// unarchived item, whatever its type — a query without a type filter. + pub types: Option>, + /// A key reads the rank of a type `next` ranks: the board carries `next`'s board, the active + /// tickets and their [`Board`], to take `next`'s candidates from at the query's `now`. + pub next: bool, + /// The graph the derived keys read (`None`: none in play). Its subjects are the listed items, + /// the active tickets when `next` is set, and its own listed items + /// ([`RankingReads::list_types`]). + pub ranking: Option, +} + +/// One listed item and the joins its `list` record and the filters read (6j6v.xkr7) — the +/// per-item input of the store-free `list`. Every field holds what the store path reads for the +/// same item: `item` with its `belongs_to` (`present_parent`), `labels` as `present_labels`, +/// `custom` every non-empty custom value, the timestamps with an unstamped `''` as `None`, +/// `contributes_to` the present targets sorted and distinct, and `parent_closed` what +/// [`parent_closed_reason`] gives. +/// +/// `#[non_exhaustive]`: build it with [`ListItem::new`] and set the fields. +#[derive(Debug, Clone, PartialEq, Eq)] +#[non_exhaustive] +pub struct ListItem { + pub item: ItemRow, + pub labels: Vec, + pub custom: BTreeMap, + pub created_at: Option, + pub updated_at: Option, + pub contributes_to: Vec, + pub parent_closed: Vec<(String, Option)>, +} + +impl ListItem { + /// An item with no joins yet. + pub fn new(item: ItemRow) -> ListItem { + ListItem { + item, + labels: Vec::new(), + custom: BTreeMap::new(), + created_at: None, + updated_at: None, + contributes_to: Vec::new(), + parent_closed: Vec::new(), + } + } + + fn joins(&self) -> RecordJoins<'_> { + RecordJoins { + labels: &self.labels, + parent_closed: &self.parent_closed, + created_at: self.created_at.as_deref(), + updated_at: self.updated_at.as_deref(), + custom: Some(&self.custom), + ..RecordJoins::default() + } + } +} + +/// The input of the store-free `list` (6j6v.xkr7): what a reader read for a [`ListReads`]. +/// +/// `#[non_exhaustive]`: build it with [`ListBoard::new`] and set the fields. +#[derive(Debug, Clone, Default)] +#[non_exhaustive] +pub struct ListBoard { + /// The types `items` holds every live, unarchived item of; `None`: every type. + pub types: Option>, + /// The listed items, by id: live and unarchived, of `types`. + pub items: BTreeMap, + /// `next`'s board, for a key that reads the rank of a type `next` ranks + /// ([`ListReads::next`]): the active tickets' rows — their labels and custom values when the + /// graph reads them — and their [`Board`]. Its own `ranking` stays `None`. + pub active: Option, + /// The graph the derived keys read ([`ListReads::ranking`]). + pub ranking: Option, +} + +impl ListBoard { + /// The listed items of `types` (`None`: every type), keyed by their ids. + pub fn new( + types: Option>, + items: impl IntoIterator, + ) -> ListBoard { + ListBoard { + types, + items: items.into_iter().map(|i| (i.item.id.clone(), i)).collect(), + active: None, + ranking: None, + } + } +} + +/// What the store-free `list` `q` reads under `cfg` (6j6v.xkr7): the listed types — `q`'s type +/// filter, else every type —, `next`'s board when a `[ranking.list]` key reads a rank `next` gives +/// and `q` has a `now`, and the graph the derived keys of a `sort = rank` read. It depends on the +/// query, not on the board. +pub fn list_reads(cfg: &PluginConfig, q: &ListQuery) -> ListReads { + derived::reads_for_list(cfg, q) +} + +/// The store's [`ListBoard`] for `reads` — the parity oracle for +/// `nxs_fold_ddb::records::list_board`, as [`active_board`] is for the records: the listed items +/// with their joins read in bulk, `next`'s board from the store's selection, and the graph. +pub fn list_board(store: &Store, reads: &ListReads) -> Result { + let items: Vec = derived::list_scope(store)? + .into_iter() + .filter(|i| { + reads + .types + .as_ref() + .is_none_or(|types| i.item_type.as_deref().is_some_and(|t| types.contains(t))) + }) + .collect(); + let ids: Vec<&str> = items.iter().map(|i| i.id.as_str()).collect(); + let mut labels = store.labels_of_bulk(&ids)?; + let mut custom = store.custom_fields_of_bulk(&ids)?; + let mut ts = store.item_timestamps_of_bulk(&ids)?; + let mut contributes = store.contributes_to_of_bulk(&ids)?; + let mut listed = Vec::with_capacity(items.len()); + for item in items { + let id = item.id.clone(); + let parent_closed = parent_closed_reason(store, &item)?; + let (created_at, updated_at) = ts.remove(&id).unwrap_or_default(); + let mut i = ListItem::new(item); + i.labels = labels.remove(&id).unwrap_or_default(); + i.custom = custom.remove(&id).unwrap_or_default(); + i.created_at = created_at; + i.updated_at = updated_at; + i.contributes_to = contributes.remove(&id).unwrap_or_default(); + i.parent_closed = parent_closed; + listed.push(i); + } + let mut board = ListBoard::new(reads.types.clone(), listed); + let graph_reads = reads.ranking.clone().unwrap_or_default(); + if reads.next { + let selection = nexus_flow_core::graph::select(store.connection())?; + let ids: Vec<&str> = selection.tickets().map(|t| t.id.as_str()).collect(); + let items = store.get_items(&ids)?; + let mut labels = if graph_reads.labels { + store.labels_of_bulk(&ids)? + } else { + BTreeMap::new() + }; + let mut custom = if graph_reads.custom { + store.custom_fields_of_bulk(&ids)? + } else { + BTreeMap::new() + }; + let tickets = items.into_iter().map(|item| { + let id = item.id.clone(); + let mut t = ActiveTicket::new(item); + t.labels = labels.remove(&id).unwrap_or_default(); + t.custom = custom.remove(&id).unwrap_or_default(); + t + }); + board.active = Some(ActiveBoard::new(selection, tickets)); + } + if let Some(r) = &reads.ranking { + let mut held: BTreeSet<&str> = board.items.keys().map(String::as_str).collect(); + if let Some(active) = &board.active { + held.extend(active.tickets.keys().map(String::as_str)); + } + let graph = ranking_graph(store, r, &held)?; + board.ranking = Some(graph); + } + Ok(board) +} + +/// [`list_page`] without a store (6j6v.xkr7): the rows of `board` that `q` keeps — its `status`, +/// its filter through the same membership rule ([`NextFilter`]) — ordered by `q.sort`, with the +/// derived keys of a `sort = rank` from the board's graph, and cut to `q.limit` with the total +/// disclosed. The same rows in the same order as [`list_page`] over a store holding the same board. +/// +/// Not paged: the store path's page cache is machine-local, and a server's paging is an open +/// decision (6j6v.15ed) — as for [`next_active_filtered`], a server cuts this list itself. So +/// `validation` for `q.paginate` or a token; and for a malformed `q.now`, a type filter (or none) +/// the board does not list every item for, and a board read for less than the ranking needs. +pub fn list_board_page(cfg: &PluginConfig, board: &ListBoard, q: &ListQuery) -> Result { + if q.paginate || q.token.is_some() { + return Err(NxfError::validation( + "the store-free `list` is not paged — a server cuts it itself; drop paginate and the \ + token", + )); + } + if let Some(now) = &q.now { + crate::validate::iso_date(now)?; + } + let listed = match &board.types { + None => true, + Some(types) => { + !q.filter.types.is_empty() && q.filter.types.iter().all(|t| types.contains(t)) + } + }; + if !listed { + return Err(NxfError::validation( + "the list board was read for other types than the query lists — read it with \ + `list_reads` for this query", + )); + } + let mut items: Vec = board + .items + .values() + // The board holds live, unarchived items; a caller-built one is held to the same rule. + .filter(|i| i.item.deleted.as_deref() != Some("1") && i.item.archived.is_none()) + .filter(|i| { + q.status + .as_deref() + .is_none_or(|s| i.item.status.as_deref() == Some(s)) + }) + .filter(|i| q.filter.keeps(&i.item, &i.labels, &i.contributes_to)) + .map(|i| i.item.clone()) + .collect(); + let key = q.sort.unwrap_or(DEFAULT_SORT_LIST); + if key == SortKey::Rank { + let derived = derived::compute_list(cfg, board, &items, q.now.as_deref())?; + let rank = TypeRank::list(cfg).with_derived(derived.as_ref()); + items.sort_by(|a, b| rank.cmp(a, b)); + } else { + order_by(cfg, &mut items, key); + } + Ok(truncate_next(items, q.limit)) +} + +/// [`list_board_page`] as the `list` JSON envelope `{"items", "total"}` — the value +/// [`Engine::list_query_value`] gives for the same board and an unpaged query, every record with +/// its joins from the item's [`ListItem`]. +/// +/// [`Engine::list_query_value`]: crate::engine::Engine::list_query_value +pub fn list_board_value(cfg: &PluginConfig, board: &ListBoard, q: &ListQuery) -> Result { + let page = list_board_page(cfg, board, q)?; + let records = Value::Array( + page.items + .iter() + .map(|i| { + let joins = board + .items + .get(&i.id) + .map(ListItem::joins) + .unwrap_or_default(); + list_record(Some(cfg), i, &joins) + }) + .collect(), + ); + Ok(next_page_envelope(records, &page, false)) +} + // ---- lane-ranked search (C6 #916.6) ---------------------------------------- // // `search` is a facade method (not CLI-only) so the CLI and an in-process consumer share ONE diff --git a/crates/facade/src/read/derived.rs b/crates/facade/src/read/derived.rs index 9906834..7c8d8ad 100644 --- a/crates/facade/src/read/derived.rs +++ b/crates/facade/src/read/derived.rs @@ -26,8 +26,18 @@ //! custom values — and runs steps 3 and 4 on the same [`Context`], so the two paths cannot evaluate //! a key apart. A neighbour counts when it is LIVE, closed and archived ones included, on both //! paths (spec §6, owner 2026-10-11). What the graph must hold is [`reads_for_next`]. - -use super::{lanes, mentioners_of, resolve_items, ActiveBoard, RankingGraph, RankingReads}; +//! +//! The store-free `list` (6j6v.xkr7, [`super::list_board_page`]) ranks a [`ListBoard`] the same +//! way: [`compute_list`] takes the listed rows from the board, `next`'s candidates — when a key +//! reads a rank `next` gives — from its active board, and every other row from its graph. A +//! `[ranking.list]` rank a key reads is computed over the type's live items, which the graph holds +//! for every type in [`RankingReads::list_types`] (the `typed` index on a server), so a `next` key +//! may read one too ([`compute_active`]). What a `list` reads is [`reads_for_list`]. + +use super::{ + lanes, mentioners_of, resolve_items, ActiveBoard, ListBoard, ListQuery, ListReads, + RankingGraph, RankingReads, SortKey, +}; use crate::error::{NxfError, Result}; use crate::plugin::{Dir, Nulls, Pick, PluginConfig, RankKey, RankOf, RankSpec, Verb}; use crate::record::{raw_field, RANKABLE_FIELDS}; @@ -138,6 +148,55 @@ pub(super) fn list_scope(store: &Store) -> Result> { .collect()) } +/// A live row and its joins, wherever a store-free board holds it: an active ticket, a listed item, +/// or a neighbour in the graph. +#[derive(Clone, Copy)] +pub(super) struct Joined<'b> { + item: &'b ItemRow, + labels: &'b [String], + custom: &'b BTreeMap, +} + +/// The row `id` of an [`ActiveBoard`] with its `graph`: an active ticket's, else a neighbour's. +fn active_joined<'b>( + board: &'b ActiveBoard, + graph: &'b RankingGraph, + id: &str, +) -> Option> { + match board.tickets.get(id) { + Some(t) => Some(Joined { + item: &t.item, + labels: &t.labels, + custom: &t.custom, + }), + None => graph.neighbours.get(id).map(|n| Joined { + item: &n.item, + labels: &n.labels, + custom: &n.custom, + }), + } +} + +/// The row `id` of a [`ListBoard`] with its `graph`: a listed item's, else an active ticket's, +/// else a neighbour's. +fn list_joined<'b>(board: &'b ListBoard, graph: &'b RankingGraph, id: &str) -> Option> { + if let Some(i) = board.items.get(id) { + return Some(Joined { + item: &i.item, + labels: &i.labels, + custom: &i.custom, + }); + } + match &board.active { + Some(active) => active_joined(active, graph, id), + None => graph.neighbours.get(id).map(|n| Joined { + item: &n.item, + labels: &n.labels, + custom: &n.custom, + }), + } +} + /// The derived side of one query, decided before any read: the type chain the ranked rows' orders /// read the rank of (leaves first), and what the derived keys in play read. struct Plan<'c> { @@ -204,6 +263,57 @@ impl Plan<'_> { fn reads_list(&self) -> bool { self.chain.iter().any(|(v, _)| *v == Verb::List) } + + /// Whether some order of the chain is a `[ranking.next]` order. + fn reads_next(&self) -> bool { + self.chain.iter().any(|(v, _)| *v == Verb::Next) + } + + /// What a [`RankingGraph`] must hold for this plan: its keys' reads, and the live items of + /// every `[ranking.list]` type of the chain the board does not list itself (`held`). + fn reads(&self, held: impl Fn(&str) -> bool) -> RankingReads { + let mut reads = self.needs.reads(); + reads.list_types = self + .chain + .iter() + .filter(|(v, ty)| *v == Verb::List && !held(ty)) + .map(|(_, ty)| ty.clone()) + .collect(); + reads + } +} + +/// The rows of a graph's listed items (the `list` scope of [`RankingReads::list_types`]), looked +/// up through `rows` — every one of them is a live row the board holds. +fn listed_rows<'b>( + graph: &RankingGraph, + rows: impl Fn(&str) -> Option>, +) -> Vec { + graph + .listed + .iter() + .filter_map(|id| rows(id).map(|j| j.item.clone())) + .collect() +} + +/// The graph a store-free query ranks with: the board's, when it was read for `needed`; an empty +/// one when the keys in play read nothing beyond the rows. `validation` otherwise — ranking without +/// those edges would order apart from the store path in silence. +fn graph_for<'g>( + graph: Option<&'g RankingGraph>, + needed: &RankingReads, + empty: &'g RankingGraph, + what: &str, +) -> Result<&'g RankingGraph> { + if *needed == RankingReads::default() { + return Ok(graph.unwrap_or(empty)); + } + graph.filter(|g| g.reads.covers(needed)).ok_or_else(|| { + NxfError::validation(format!( + "{what} ranks by derived ranking keys, and the board was read without the graph they \ + read — fill its `ranking` for the reads its plugin names" + )) + }) } /// Every row a key is evaluated on: the ranked rows and the rows of the chain's scopes. @@ -268,12 +378,13 @@ pub(super) fn compute( /// [`compute`] for the store-free `next` (6j6v.1c5s): the values the derived keys of /// `[ranking.next]` rank `candidates` — every candidate of the board — by, read from the /// [`RankingGraph`] a reader filled instead of from a store. The candidates are `next`'s scope for -/// "rank", as on the store path. `None` when the policy has no derived key. +/// "rank", as on the store path; the `list` scope of a `[ranking.list]` type the chain reaches is +/// the graph's listed items (6j6v.xkr7). `None` when the policy has no derived key. /// /// The same [`Context`] evaluates the keys as on the store path; only where its edges, rows, -/// labels and custom values come from differs ([`Context::read_graph`]). A chain that reaches a -/// `[ranking.list]` order is refused before this is called ([`reads_for_next`]); it answers -/// `validation` here too rather than rank without that scope. +/// labels and custom values come from differs ([`Context::read_graph`]). `validation` when the graph +/// was read for less than the plan needs — a `[ranking.list]` type of the chain missing from +/// [`RankingReads::list_types`] among it. pub(super) fn compute_active( cfg: &PluginConfig, board: &ActiveBoard, @@ -283,84 +394,179 @@ pub(super) fn compute_active( let Some(plan) = plan(cfg, Verb::Next, candidates) else { return Ok(None); }; - if plan.reads_list() { - return Err(list_scope_refusal(cfg)); - } - let node_rows = plan.node_rows(candidates, &[]); + let needed = plan.reads(|_| false); + let graph = graph_for( + Some(graph), + &needed, + graph, + &format!("plugin '{}' (`next`)", cfg.name), + )?; + let rows = |id: &str| active_joined(board, graph, id); + let listed = listed_rows(graph, rows); + let node_rows = plan.node_rows(candidates, &listed); let mut ctx = Context::new(cfg, subjects(candidates, &node_rows)); - ctx.read_graph(board, graph, &plan.needs); + ctx.read_graph(rows, graph, &plan.needs); Ok(Some(ctx.finish(&plan, Verb::Next, candidates, &node_rows))) } -/// What the store-free `next` of `cfg` reads beyond the active tickets' records (6j6v.1c5s): -/// `None` when `[ranking.next]` declares no derived key. Gathered over every order of the section, -/// so it does not depend on which types a board holds. +/// [`compute`] for the store-free `list` (6j6v.xkr7): the values the derived keys of +/// `[ranking.list]` rank `ranked` by, from a [`ListBoard`]. The `list` scope of a type is its live, +/// unarchived items — the board's listed items for the types it lists, the graph's listed items +/// for the others; `next`'s scope, when the chain reaches a `[ranking.next]` order, is the +/// candidates of the board's active board at `now`. `None` when the policy has no derived key. /// -/// `validation` when an order of `[ranking.next]` reads — directly or down its type chain — the -/// rank of a type only `[ranking.list]` orders (spec §12, first point): that rank is computed over -/// the live items of the type, and a server reads only the active ones (owner, 2026-10-11). +/// `validation` as on the store path when the chain reads a `next` rank and `now` is missing, and +/// when the board was read for less than the plan needs: no active board for such a chain, a +/// graph read for fewer reads, a chain type neither listed nor in the graph. +pub(super) fn compute_list( + cfg: &PluginConfig, + board: &ListBoard, + ranked: &[ItemRow], + now: Option<&str>, +) -> Result> { + let Some(plan) = plan(cfg, Verb::List, ranked) else { + return Ok(None); + }; + let held = |ty: &str| board.types.as_ref().is_none_or(|t| t.contains(ty)); + let needed = plan.reads(held); + let empty = RankingGraph::default(); + let graph = graph_for( + board.ranking.as_ref(), + &needed, + &empty, + &format!("plugin '{}' (`list`)", cfg.name), + )?; + let next_owned; + let next_rows: &[ItemRow] = if plan.reads_next() { + let now = now.ok_or_else(|| { + NxfError::validation( + "the plugin's [ranking.list] reads the rank of a type that `next` ranks, which is \ + computed over `next`'s candidates at a reference time — pass `now`", + ) + })?; + let active = board.active.as_ref().ok_or_else(|| { + NxfError::validation(format!( + "plugin '{}' ranks `list` by the rank of a type that `next` ranks, and the board \ + was read without the active board that rank is computed over — read it with \ + `ListReads::next`", + cfg.name + )) + })?; + let candidates = active.board.lanes(now).candidates; + next_owned = active.items(candidates.iter().map(|c| &c.id)); + &next_owned + } else { + &[] + }; + let rows = |id: &str| list_joined(board, graph, id); + let list_owned: Vec = if plan.reads_list() { + let mut scope: Vec = board.items.values().map(|i| i.item.clone()).collect(); + scope.extend( + listed_rows(graph, rows) + .into_iter() + .filter(|i| !board.items.contains_key(&i.id)), + ); + scope + } else { + Vec::new() + }; + let node_rows = plan.node_rows(next_rows, &list_owned); + let mut ctx = Context::new(cfg, subjects(ranked, &node_rows)); + ctx.read_graph(rows, graph, &plan.needs); + Ok(Some(ctx.finish(&plan, Verb::List, ranked, &node_rows))) +} + +/// What the store-free `next` of `cfg` reads beyond the active tickets' records (6j6v.1c5s): +/// `None` when `[ranking.next]` declares no derived key. Gathered over every order of the section +/// and of its type chain, so it does not depend on which types a board holds. Since 6j6v.xkr7 an +/// order that reads the rank of a type only `[ranking.list]` orders is no longer refused: the +/// type lands in [`RankingReads::list_types`], whose live items a reader puts in the graph. pub(super) fn reads_for_next(cfg: &PluginConfig) -> Result> { - let ranking = &cfg.ranking; - let spec = ranking.spec(Verb::Next); + let spec = cfg.ranking.spec(Verb::Next); if !spec.has_derived() { return Ok(None); } + let orders: Vec<&[RankKey]> = std::iter::once(spec.order.as_slice()) + .chain(spec.by_type.values().map(|o| o.order.as_slice())) + .collect(); + // `next`'s own per-type orders start the chain: an order there may read another's rank. + let starts = spec.by_type.keys().map(|ty| (Verb::Next, ty.clone())); + Ok(Some(chain_reads(cfg, &orders, starts, |_| false).0)) +} + +/// What the store-free `list` `q` asks for reads (6j6v.xkr7): its listed types (`None`: every +/// type), whether `next`'s board is needed, and the graph. Only `sort = rank` ranks by the +/// derived keys; any other order reads the listed items alone. +pub(super) fn reads_for_list(cfg: &PluginConfig, q: &ListQuery) -> ListReads { + let types: Option> = + (!q.filter.types.is_empty()).then(|| q.filter.types.iter().cloned().collect()); + let mut reads = ListReads { + types: types.clone(), + next: false, + ranking: None, + }; + let spec = cfg.ranking.spec(Verb::List); + if q.sort != Some(SortKey::Rank) || !spec.has_derived() { + return reads; + } + let orders: Vec<&[RankKey]> = match &types { + Some(types) => types + .iter() + .map(|ty| spec.order_for(Some(ty.as_str()))) + .collect(), + None => std::iter::once(spec.order.as_slice()) + .chain(spec.by_type.values().map(|o| o.order.as_slice())) + .collect(), + }; + let starts: Vec = orders + .iter() + .flat_map(|keys| cfg.ranking.rank_deps(keys, Verb::List)) + .collect(); + let held = |ty: &str| types.as_ref().is_none_or(|t| t.contains(ty)); + let (graph, reads_next) = chain_reads(cfg, &orders, starts, held); + // Without `now` a chain that reads a `next` rank answers `validation` where a ranked row + // needs it, as on the store path — reading `next`'s board for it would be wasted. + reads.next = reads_next && q.now.is_some(); + if graph != RankingReads::default() { + reads.ranking = Some(graph); + } + reads +} + +/// The reads of `orders` and of every order down the type chain from `starts`, the chain's +/// `[ranking.list]` types not `held` among them, and whether that chain holds a `[ranking.next]` +/// order. +fn chain_reads( + cfg: &PluginConfig, + orders: &[&[RankKey]], + starts: impl IntoIterator, + held: impl Fn(&str) -> bool, +) -> (RankingReads, bool) { let mut chain: Vec = Vec::new(); let mut seen: BTreeSet = BTreeSet::new(); - for ty in spec.by_type.keys() { - post_order(cfg, (Verb::Next, ty.clone()), &mut seen, &mut chain); - } - if chain.iter().any(|(v, _)| *v == Verb::List) { - return Err(list_scope_refusal(cfg)); + for node in starts { + post_order(cfg, node, &mut seen, &mut chain); } - let keys: Vec<&RankKey> = spec - .order + let mut keys: Vec<&RankKey> = orders .iter() - .chain(spec.by_type.values().flat_map(|o| o.order.iter())) + .flat_map(|o| o.iter()) .filter(|k| k.is_derived()) .collect(); - let needs = Needs::of(&keys); - let mut reads = RankingReads::default(); - reads.kinds.extend(needs.kinds.iter().copied()); - if needs.mentioned_by { - reads.kinds.insert(EdgeKind::Mentions); + for (v, ty) in &chain { + keys.extend( + cfg.ranking.spec(*v).by_type[ty] + .order + .iter() + .filter(|k| k.is_derived()), + ); } - reads.labels = needs.labels; - reads.custom = needs.custom; - Ok(Some(reads)) -} - -/// The refusal of a `[ranking.next]` whose type chain reaches a `[ranking.list]` order, on the -/// store-free path. -fn list_scope_refusal(cfg: &PluginConfig) -> NxfError { - let ranking = &cfg.ranking; - let reached: Vec = ranking - .spec(Verb::Next) - .by_type - .keys() - .flat_map(|ty| { - let mut chain = Vec::new(); - post_order( - cfg, - (Verb::Next, ty.clone()), - &mut BTreeSet::new(), - &mut chain, - ); - chain - .into_iter() - .filter(|(v, _)| *v == Verb::List) - .map(move |(_, list_ty)| { - format!("[ranking.next.{ty}] reads the rank of '{list_ty}'") - }) - }) + let mut reads = Needs::of(&keys).reads(); + reads.list_types = chain + .iter() + .filter(|(v, ty)| *v == Verb::List && !held(ty)) + .map(|(_, ty)| ty.clone()) .collect(); - NxfError::validation(format!( - "plugin '{}' ranks `next` by the rank of a type only [ranking.list] orders ({}): that rank \ - is computed over the type's live items, which the store-free lanes do not hold — answer \ - `next` from a store", - cfg.name, - reached.join("; ") - )) + (reads, chain.iter().any(|(v, _)| *v == Verb::Next)) } /// Depth-first post-order over the type chain: every order a node reads the rank of lands before @@ -419,6 +625,21 @@ impl Needs { } } +impl Needs { + /// These needs as the [`RankingReads`] of a graph: `mentions` among the kinds when a count reads + /// `mentioned_by`. No list types; the caller adds them. + fn reads(&self) -> RankingReads { + let mut reads = RankingReads::default(); + reads.kinds.extend(self.kinds.iter().copied()); + if self.mentioned_by { + reads.kinds.insert(EdgeKind::Mentions); + } + reads.labels = self.labels; + reads.custom = self.custom; + reads + } +} + /// One count, from a `count` key or a `sum` summand. struct Count<'a> { kind: &'a str, @@ -541,11 +762,16 @@ impl<'a> Context<'a> { /// The reads from a [`RankingGraph`] instead of a store (6j6v.1c5s): the same adjacency, reverse /// mentions, neighbour rows, labels and custom values [`read_edges`](Self::read_edges) reads, - /// taken from the graph's edges and from the board's tickets and the graph's neighbours. The - /// graph holds every edge of the kinds read that touches an active ticket with both ends live, - /// `parent` as the `present_parent` projection — the edges the store's pass yields for subjects - /// that are all active tickets, as `next`'s candidates are. - fn read_graph(&mut self, board: &ActiveBoard, graph: &RankingGraph, needs: &Needs) { + /// taken from the graph's edges and, through `rows`, from the board's rows — its active + /// tickets, its listed items — and the graph's neighbours. The graph holds every edge of the + /// kinds read that touches a subject with both ends live, `parent` as the `present_parent` + /// projection — the edges the store's pass yields for the same subjects. + fn read_graph<'b>( + &mut self, + rows: impl Fn(&str) -> Option>, + graph: &RankingGraph, + needs: &Needs, + ) { let mut mention_sources: BTreeMap> = BTreeMap::new(); for e in &graph.edges { if needs.kinds.contains(&e.kind) { @@ -572,21 +798,13 @@ impl<'a> Context<'a> { .insert(e.from.as_str()); } } - // A row of the board: an active ticket's, or a live neighbour's. - let board_row = |id: &str| -> Option<&ItemRow> { - board - .tickets - .get(id) - .map(|t| &t.item) - .or_else(|| graph.neighbours.get(id).map(|n| &n.item)) - }; for (target, froms) in mention_sources { - let rows: Vec = froms + let found: Vec = froms .into_iter() - .filter_map(|f| board_row(f).cloned()) + .filter_map(|f| rows(f).map(|j| j.item.clone())) .collect(); - if !rows.is_empty() { - self.mentioners.insert(target, rows); + if !found.is_empty() { + self.mentioners.insert(target, found); } } let missing: BTreeSet<&str> = self @@ -599,23 +817,19 @@ impl<'a> Context<'a> { .collect(); self.neighbours = missing .into_iter() - .filter_map(|id| board_row(id).map(|r| (id.to_string(), r.clone()))) + .filter_map(|id| rows(id).map(|j| (id.to_string(), j.item.clone()))) .collect(); if needs.labels || needs.custom { let mut ids: BTreeSet<&str> = self.neighbours.keys().map(String::as_str).collect(); ids.extend(self.subjects.keys().copied()); ids.extend(self.mentioners.values().flatten().map(|i| i.id.as_str())); for id in ids { - let (labels, custom) = match (board.tickets.get(id), graph.neighbours.get(id)) { - (Some(t), _) => (&t.labels, &t.custom), - (None, Some(n)) => (&n.labels, &n.custom), - (None, None) => continue, - }; - if needs.labels && !labels.is_empty() { - self.labels.insert(id.to_string(), labels.clone()); + let Some(j) = rows(id) else { continue }; + if needs.labels && !j.labels.is_empty() { + self.labels.insert(id.to_string(), j.labels.to_vec()); } - if needs.custom && !custom.is_empty() { - self.custom.insert(id.to_string(), custom.clone()); + if needs.custom && !j.custom.is_empty() { + self.custom.insert(id.to_string(), j.custom.clone()); } } } diff --git a/crates/facade/tests/derived_ranking.rs b/crates/facade/tests/derived_ranking.rs index cdb778c..c49f900 100644 --- a/crates/facade/tests/derived_ranking.rs +++ b/crates/facade/tests/derived_ranking.rs @@ -11,6 +11,8 @@ //! - Determinism (§9): another edge insertion order ranks alike, and ties fall to the id. //! - Bounded cost (§9, a query-shape guard in the style of xn8s): the edge pass runs once per query //! however many candidates there are. +//! - Without a store (6j6v.1c5s, 6j6v.xkr7): `read::next_active` and `read::list_board_page` over +//! the boards a server reads rank as the store path does. //! //! Its own test binary, so the fixture plugin does not pollute the facade lib tests' "exactly two //! OSS plugins" assertion. @@ -521,11 +523,12 @@ fn a_closed_archived_or_deleted_neighbour_reads_alike_with_and_without_a_store() } #[test] -fn a_next_order_reading_the_rank_of_a_type_only_list_orders_is_refused_without_a_store() { +fn a_next_order_reading_the_rank_of_a_type_only_list_orders_ranks_from_the_listed_items() { // Spec §12 point 1 lets a `next` key read the rank `[ranking.list]` gives a type. That rank is - // computed over the type's live items, which a board of active tickets does not hold: the - // store-free `next` refuses the plugin (owner, 2026-10-11), the store path ranks it. Any other - // derived `next` key ranks without a store. + // computed over the type's live, unarchived items, which a board of active tickets does not + // hold: since 6j6v.xkr7 the reads name the type (`list_types`) and the graph holds its items — + // a server reads them off its `typed` index — so the store-free `next` ranks as the store + // path does. A board read without them is refused. let cfg: plugin::PluginConfig = toml::from_str( r#" name = "list-rank-fixture" @@ -533,7 +536,7 @@ fn a_next_order_reading_the_rank_of_a_type_only_list_orders_is_refused_without_a en = "x" de = "y" [priority] - labels = ["P0", "P1"] + labels = ["P0", "P1", "P2"] [types] list = ["action", "person"] [vocabulary.status] @@ -545,33 +548,188 @@ fn a_next_order_reading_the_rank_of_a_type_only_list_orders_is_refused_without_a [ranking.next.action] order = [ { related = "mentions", type = "person", of = "rank", nulls = "last" } ] [ranking.list.person] - order = [ { field = "priority", dir = "asc" } ] + order = [ { count = "mentioned_by", dir = "desc" }, { field = "priority", dir = "asc" } ] [presentation.list] columns = ["id"] "#, ) .unwrap(); let mut s = Store::open_in_memory(1); - item(&mut s, "zz12.a1", "action", "1"); - item(&mut s, "zz12.u1", "person", "0"); - edge(&mut s, "zz12.a1", "zz12.u1", EdgeKind::Mentions); - let err = read::next_ranking_reads(&cfg).unwrap_err(); - assert_eq!(err.kind, ErrorKind::Validation); - assert!( - err.msg - .contains("[ranking.next.action] reads the rank of 'person'"), - "{}", - err.msg + // Persons: u1 mentioned twice (rank 0); u2 and the closed u3 once each, u3 with the better + // priority; u4 archived (no rank: outside the list scope); u5 deleted. + for (u, prio) in [ + ("u1", "2"), + ("u2", "1"), + ("u3", "0"), + ("u4", "0"), + ("u5", "0"), + ] { + item(&mut s, &format!("zz12.{u}"), "person", prio); + } + close(&mut s, "zz12.u3"); + s.set_field("zz12.u4", "archived", Some("2026-06-01".into()), "t"); + for (a, person) in [ + ("zz12.a1", "zz12.u2"), + ("zz12.a2", "zz12.u1"), + ("zz12.a3", "zz12.u3"), + ("zz12.a4", "zz12.u4"), + ("zz12.a5", "zz12.u5"), + ] { + item(&mut s, a, "action", "1"); + edge(&mut s, a, person, EdgeKind::Mentions); + } + item(&mut s, "zz12.a6", "action", "1"); + edge(&mut s, "zz12.a6", "zz12.u1", EdgeKind::Mentions); + s.delete_item("zz12.u5", "t"); + + let reads = read::next_ranking_reads(&cfg) + .unwrap() + .expect("derived next keys"); + assert_eq!( + reads.list_types.iter().collect::>(), + ["person"], + "{reads:?}" ); - let board = read::active_board(&s).unwrap(); + let board = read::with_ranking(&s, read::active_board(&s).unwrap(), &reads).unwrap(); + let graph = board.ranking.as_ref().unwrap(); + assert_eq!( + graph.listed.iter().collect::>(), + ["zz12.u1", "zz12.u2", "zz12.u3"], + "the live, unarchived persons" + ); + let store = read::next(&cfg, &s, NOW, None).unwrap(); + assert_eq!( + of_type(&store, "action"), + ["zz12.a2", "zz12.a6", "zz12.a3", "zz12.a1", "zz12.a4", "zz12.a5"] + ); + assert_eq!( + ids(&read::next_active(&cfg, &board, NOW).unwrap()), + ids(&store) + ); + + // Without the person's items the graph is short of what the keys read: refused. + let mut short = reads.clone(); + short.list_types.clear(); + let board = read::with_ranking(&s, read::active_board(&s).unwrap(), &short).unwrap(); assert_eq!( read::next_active(&cfg, &board, NOW).unwrap_err().kind, ErrorKind::Validation ); +} + +#[test] +fn the_store_free_list_ranks_persons_as_list_query_does() { + // 6j6v.xkr7: the hosted `list` — `read::list_board_page` over a `ListBoard` read for + // `read::list_reads` — gives the rows, the order, the total and the JSON `Engine::list_query` + // gives: persons by a sum of mentions, then by the rank `next` gives the projects they + // contribute to (`now` takes `next`'s candidates), filtered by type, status, container and + // label as locally. + let tmp = workspace("derived-rank-fixture"); + spec_board(tmp.path()); + let cfg = plugin::load("derived-rank-fixture").unwrap(); + let s = writer(tmp.path()); + let engine = engine(&tmp); + let queries = [ + ListQuery::new() + .sort(SortKey::Rank) + .filter(NextFilter::new().with_type("person")) + .now(NOW), + ListQuery::new() + .sort(SortKey::Rank) + .filter(NextFilter::new().with_type("person")) + .now(NOW) + .limit(2), + ListQuery::new().sort(SortKey::Rank).now(NOW), + ListQuery::new() + .sort(SortKey::Rank) + .filter( + NextFilter::new() + .with_type("action") + .with_container("zz12.p5"), + ) + .now(NOW), + ListQuery::new() + .sort(SortKey::Rank) + .filter(NextFilter::new().with_label("waiting")) + .now(NOW), + ListQuery::new().status("closed"), + ListQuery::new() + .filter(NextFilter::new().with_type("project").with_type("goal")) + .sort(SortKey::Rank), + ]; + for q in &queries { + let reads = read::list_reads(&cfg, q); + let board = read::list_board(&s, &reads).unwrap(); + let page = read::list_board_page(&cfg, &board, q).unwrap(); + let want = engine.list_query(q).unwrap(); + assert_eq!(ids(&page.items), ids(&want.items), "{q:?}"); + assert_eq!(page.total, want.total, "{q:?}"); + assert_eq!( + read::list_board_value(&cfg, &board, q).unwrap(), + engine.list_query_value(q).unwrap(), + "{q:?}" + ); + } + let persons = read::list_board_page( + &cfg, + &read::list_board(&s, &read::list_reads(&cfg, &queries[0])).unwrap(), + &queries[0], + ) + .unwrap(); + assert_eq!(ids(&persons.items), PERSONS); + let reads = read::list_reads(&cfg, &queries[0]); assert!( - read::next(&cfg, &s, NOW, None).is_ok(), - "the store path ranks it" + reads.next, + "persons read the rank `next` gives projects: {reads:?}" ); + assert_eq!( + reads.types, + Some(["person".to_string()].into_iter().collect()) + ); +} + +#[test] +fn the_store_free_list_refuses_what_its_board_was_not_read_for() { + let tmp = workspace("derived-rank-fixture"); + spec_board(tmp.path()); + let cfg = plugin::load("derived-rank-fixture").unwrap(); + let s = writer(tmp.path()); + let persons = ListQuery::new() + .sort(SortKey::Rank) + .filter(NextFilter::new().with_type("person")) + .now(NOW); + let board = read::list_board(&s, &read::list_reads(&cfg, &persons)).unwrap(); + let refused = |q: &ListQuery, board: &read::ListBoard| { + let err = read::list_board_page(&cfg, board, q).unwrap_err(); + assert_eq!(err.kind, ErrorKind::Validation, "{q:?}"); + err.msg + }; + // Not paged: a server cuts the list itself. + refused(&persons.clone().limit(2).paginate(), &board); + // Other types, or every type, than the board lists. + refused( + &ListQuery::new().filter(NextFilter::new().with_type("project")), + &board, + ); + refused(&ListQuery::new(), &board); + // Without `now`, `next`'s candidates cannot be taken — as on the store path. + let no_now = ListQuery::new() + .sort(SortKey::Rank) + .filter(NextFilter::new().with_type("person")); + let reads = read::list_reads(&cfg, &no_now); + assert!(!reads.next, "nothing to read `next`'s board for: {reads:?}"); + let msg = refused(&no_now, &read::list_board(&s, &reads).unwrap()); + assert!(msg.contains("pass `now`"), "{msg}"); + // A board without the graph its keys read. + let mut short = read::list_reads(&cfg, &persons); + short.ranking = None; + let msg = refused(&persons, &read::list_board(&s, &short).unwrap()); + assert!(msg.contains("derived ranking keys"), "{msg}"); + // Without `next`'s board. + let mut short = read::list_reads(&cfg, &persons); + short.next = false; + let msg = refused(&persons, &read::list_board(&s, &short).unwrap()); + assert!(msg.contains("active board"), "{msg}"); } // ---- each kind over hand-built boards (§9 "per kind") ------------------------------------- diff --git a/crates/fold-ddb/src/board.rs b/crates/fold-ddb/src/board.rs index 36aea0e..d43f5d0 100644 --- a/crates/fold-ddb/src/board.rs +++ b/crates/fold-ddb/src/board.rs @@ -1,5 +1,5 @@ -//! The board's two indexes, kept beside the folded rows, and the reads that use them -//! (6j6v.vvw6 points 4 and 5). +//! The board's indexes, kept beside the folded rows, and the reads that use them +//! (6j6v.vvw6 points 4 and 5; `typed`, 6j6v.xkr7). //! //! # `active` //! @@ -27,6 +27,17 @@ //! one instant keep one order. A ticket without the instant sits under `"0"`, which sorts after //! every date — last, as the local lanes put it. [`dated`] reads one lane newest first. //! +//! # `typed` +//! +//! Sparse as well (6j6v.xkr7): a live item — not deleted, archived or not — with a type sits in +//! partition `#` ([`typed_partition`]) under its own sort key, so ONE Query lists the +//! live items of a type in id order: the `list` scope of a type the derived ranking keys rank by +//! (docs/specs/derived-ranking-keys.md §6). Like the `active` flag of a ticket it is a function of +//! the row alone, set whenever an op touches the row ([`refresh_item`]) — after the row, so a run +//! that dies in between sets it when it folds the op again. Generic: every typed item is listed, +//! whatever the plugin ranks, at one attribute per item row. A row's `typed` attribute never +//! names another row, so a retype or a delete moves nothing else. +//! //! # Who owns a label or a thread link //! //! `label_adds` and `thread_link_adds` key on the add's tag alone, so nothing in them is found by @@ -72,7 +83,8 @@ use crate::layout::{ adjacency_key, adjacency_prefix, contributes_adjacency_key, label_adjacency_key, - link_adjacency_key, row_key, table_prefix, ACTIVE, DATED, DATED_AT, MAX_KEY_BYTES, SK, + link_adjacency_key, row_key, table_prefix, typed_partition, ACTIVE, DATED, DATED_AT, + MAX_KEY_BYTES, SK, TYPED, }; use crate::table::{text, Dated, Row, Table}; use crate::write::{Cond, Write}; @@ -233,6 +245,29 @@ pub fn is_active(row: &Row) -> bool { ) } +/// The type a stored `items` row is listed under on the `typed` index (6j6v.xkr7): its type, when +/// it has one and is live — not deleted; archived or not. A type too long for an index partition +/// ([`MAX_PARTITION_BYTES`](crate::layout::MAX_PARTITION_BYTES)) is never folded: [`typed_partitions`] refuses its op first. +pub fn typed(row: &Row) -> Option<&str> { + if text(row, "deleted") == Some("1") { + return None; + } + text(row, "type") +} + +/// The `typed` partitions the item rows these changes write would be listed under — what +/// [`Folder`](crate::fold::Folder) checks against +/// [`MAX_PARTITION_BYTES`](crate::layout::MAX_PARTITION_BYTES) before an op writes +/// anything. +pub(crate) fn typed_partitions(stream: &str, changes: &[Change]) -> Vec { + changes + .iter() + .filter(|c| c.table == "items") + .filter_map(|c| effect_text(c, "type")) + .map(|ty| typed_partition(stream, ty)) + .collect() +} + /// The `dated` lane of a stored `items` row, and the instant it sorts by. pub fn lane(row: &Row) -> Option<(Dated, String)> { if text(row, "deleted") == Some("1") { @@ -331,6 +366,14 @@ pub(crate) async fn refresh_item(table: &T, id: &str) -> Result<(), T: remove.push(ACTIVE.to_string()); } } + let typed_now = typed(&row).map(|ty| typed_partition(table.stream(), ty)); + let typed_was = text(&row, TYPED).map(str::to_string); + if typed_now != typed_was { + match typed_now { + Some(p) => set.push((TYPED.to_string(), Cell::Text(p))), + None => remove.push(TYPED.to_string()), + } + } let dated_now = lane(&row).map(|(l, at)| (l.partition(table.stream()), dated_key(&at, id))); let dated_was = match (row.get(DATED), row.get(DATED_AT)) { (Some(Cell::Text(p)), Some(Cell::Text(at))) => Some((p.clone(), at.clone())), diff --git a/crates/fold-ddb/src/ddb.rs b/crates/fold-ddb/src/ddb.rs index 6089713..323d715 100644 --- a/crates/fold-ddb/src/ddb.rs +++ b/crates/fold-ddb/src/ddb.rs @@ -1,14 +1,16 @@ //! [`DynamoDbTable`]: one stream's partition of the views table in DynamoDB, and [`create_table`], //! the table the contract in the crate doc describes. //! -//! Every request names the stream's partition key; the board's two reads name an index. There is +//! Every request names the stream's partition key; the board's three index reads name an index. There is //! no Scan anywhere in this file, and the `dynamodb-local` CI job records every request a fold //! makes to prove it stays that way. //! //! The adapter is async and runs on the caller's runtime — a fold run in a Lambda awaits it //! directly, unlike the relay's op store, which bridges the SDK into its synchronous traits. -use crate::layout::{ACTIVE, ACTIVE_INDEX, DATED, DATED_AT, DATED_INDEX, PK, SK}; +use crate::layout::{ + typed_partition, ACTIVE, ACTIVE_INDEX, DATED, DATED_AT, DATED_INDEX, PK, SK, TYPED, TYPED_INDEX, +}; use crate::table::{Dated, Outcome, Row, Table}; use crate::write::Write; use aws_sdk_dynamodb::error::{DisplayErrorContext, ProvideErrorMetadata}; @@ -37,8 +39,8 @@ fn err(e: E) -> DdbError { DdbError(DisplayErrorContext(e).to_string()) } -/// The views table of `name`, as the deploying stack must provision it: keys, both sparse -/// indexes projecting every attribute, on-demand billing. Production code never sends it; the +/// The views table of `name`, as the deploying stack must provision it: keys, the three sparse +/// indexes (`active`, `dated`, `typed`) projecting every attribute, on-demand billing. Production code never sends it; the /// integration tests do, and a stack is checked against it. pub fn create_table(client: &Client, name: &str) -> CreateTableFluentBuilder { let s = |n: &str| { @@ -77,10 +79,12 @@ pub fn create_table(client: &Client, name: &str) -> CreateTableFluentBuilder { .attribute_definitions(s(ACTIVE)) .attribute_definitions(s(DATED)) .attribute_definitions(s(DATED_AT)) + .attribute_definitions(s(TYPED)) .key_schema(key(PK, KeyType::Hash)) .key_schema(key(SK, KeyType::Range)) .global_secondary_indexes(index(ACTIVE_INDEX, ACTIVE, SK)) .global_secondary_indexes(index(DATED_INDEX, DATED, DATED_AT)) + .global_secondary_indexes(index(TYPED_INDEX, TYPED, SK)) } /// One stream's partition of the views table `table`. @@ -303,4 +307,19 @@ impl Table for DynamoDbTable { ) .await } + + async fn query_typed(&self, item_type: &str) -> Result, DdbError> { + self.query_all( + Some(TYPED_INDEX), + "#t = :t", + &[("#t", TYPED)], + &[( + ":t", + AttributeValue::S(typed_partition(&self.stream, item_type)), + )], + false, + None, + ) + .await + } } diff --git a/crates/fold-ddb/src/fold.rs b/crates/fold-ddb/src/fold.rs index e4f8075..9770de4 100644 --- a/crates/fold-ddb/src/fold.rs +++ b/crates/fold-ddb/src/fold.rs @@ -23,7 +23,9 @@ //! so its watermark names no index revision at all and the next run refolds it. Revision 2 //! (6j6v.15ed) adds the `.cadj#` entries, a ticket's outgoing `contributes_to` edges. Revision 3 //! (6j6v.1c5s) makes the `.adj#` entries self-describing — ends, kind, coordinate, presence — for -//! the derived ranking keys ([`crate::records::with_ranking`]). +//! the derived ranking keys ([`crate::records::with_ranking`]), and sets the `typed` index +//! attribute on every live item row (6j6v.xkr7, [`crate::records::list_board`]). Both ship in one +//! release: no stream was ever folded by revision 3 without the attribute. //! //! # What the server cannot see //! @@ -34,7 +36,9 @@ //! coordinate. use crate::board; -use crate::layout::{Layout, LayoutError, MAX_ITEM_BYTES, MAX_KEY_BYTES, WATERMARK_KEY}; +use crate::layout::{ + Layout, LayoutError, MAX_ITEM_BYTES, MAX_KEY_BYTES, MAX_PARTITION_BYTES, WATERMARK_KEY, +}; use crate::table::{Outcome, Table}; use crate::write::{plan, Cond, Write}; use nxs_foundation::change::{Cell, Change, Wins}; @@ -49,7 +53,7 @@ pub const INDEX_DOMAIN: &str = "index"; /// The revision of those rules. 1: the entries under a ticket for its label and thread link adds /// (6j6v.t1ym). 2: the entries under a ticket for its outgoing `contributes_to` edges (6j6v.15ed). /// 3: the adjacency entries copy their edge's ends, kind and coordinate and say whether it is -/// removed (6j6v.1c5s). +/// removed (6j6v.1c5s), and every live item row carries its `typed` index partition (6j6v.xkr7). pub const INDEX_REVISION: i64 = 3; /// The platform's reducers — the board's, the chat's and the memory's. @@ -162,7 +166,8 @@ impl Folder { /// Fold one op: carry out its changes, then keep the board's indexes. /// - /// An op the table cannot hold — a key past `MAX_KEY_BYTES`, a row past `MAX_ITEM_BYTES` — is + /// An op the table cannot hold — a key past `MAX_KEY_BYTES`, a type whose `typed` index + /// partition is past `MAX_PARTITION_BYTES`, a row past `MAX_ITEM_BYTES` — is /// [`Folded::Refused`] before it writes anything, rather than failing the batch for ever: the /// relay keeps it, as it keeps any op a server does not fold. A row that only outgrows the limit /// through earlier ops refuses the one write that would break it ([`Outcome::TooLarge`]); the @@ -197,6 +202,16 @@ impl Folder { ))); } } + if let Some(p) = board::typed_partitions(table.stream(), &changes) + .into_iter() + .find(|p| p.len() > MAX_PARTITION_BYTES) + { + return Ok(refuse(format!( + "a type whose `typed` index partition has {} bytes, past the {MAX_PARTITION_BYTES} \ + a partition key may have", + p.len() + ))); + } if let Some(w) = writes.iter().find(|w| w.bytes() > MAX_ITEM_BYTES) { return Ok(refuse(format!( "a row of {} bytes, past the {MAX_ITEM_BYTES} an item may have", diff --git a/crates/fold-ddb/src/layout.rs b/crates/fold-ddb/src/layout.rs index ffd4943..73475e6 100644 --- a/crates/fold-ddb/src/layout.rs +++ b/crates/fold-ddb/src/layout.rs @@ -26,18 +26,26 @@ pub const ACTIVE: &str = "nxf_active"; pub const DATED: &str = "nxf_dated"; /// The sort key of the `dated` index: the instant the ticket entered its lane. pub const DATED_AT: &str = "nxf_dated_at"; +/// The partition key of the `typed` index (6j6v.xkr7): `#` ([`typed_partition`]), +/// present only on a live item row — not deleted, archived or not — that has a type. +pub const TYPED: &str = "nxf_typed"; /// The `active` index. pub const ACTIVE_INDEX: &str = "active"; /// The `dated` index. pub const DATED_INDEX: &str = "dated"; +/// The `typed` index (6j6v.xkr7). +pub const TYPED_INDEX: &str = "typed"; /// The longest sort key — and `dated` index sort key — DynamoDB stores, in bytes. pub const MAX_KEY_BYTES: usize = 1024; +/// The longest partition key value DynamoDB stores — on the table or on an index — in bytes. Only +/// the `typed` index's can come near it: it carries an item's type ([`typed_partition`]). +pub const MAX_PARTITION_BYTES: usize = 2048; /// The largest item DynamoDB stores, in bytes, counting attribute names and values. pub const MAX_ITEM_BYTES: usize = 400 * 1024; /// Attribute names no view column may take: the table's own keys and the index attributes. -pub const RESERVED: [&str; 5] = [PK, SK, ACTIVE, DATED, DATED_AT]; +pub const RESERVED: [&str; 6] = [PK, SK, ACTIVE, DATED, DATED_AT, TYPED]; /// The fold's own bookkeeping, outside every view table: a view table's name is an identifier and /// cannot start with `.`. @@ -69,6 +77,13 @@ pub fn row_key(table: &str, key: &[(String, Cell)]) -> Result#`, the +/// type with `%` and `#` escaped as a key value is, so two streams' types can never share a +/// partition (`a` + `b#c` and `a#b` + `c`). +pub fn typed_partition(stream: &str, item_type: &str) -> String { + format!("{stream}#{}", escape(item_type)) +} + /// The prefix every row of `table` starts with. pub fn table_prefix(table: &str) -> String { format!("{table}#") diff --git a/crates/fold-ddb/src/lib.rs b/crates/fold-ddb/src/lib.rs index f97134f..05e279d 100644 --- a/crates/fold-ddb/src/lib.rs +++ b/crates/fold-ddb/src/lib.rs @@ -1,7 +1,8 @@ //! Folding on a server (6j6v.vvw6): the reducers' [described changes](nxs_foundation::change) //! carried out as conditional writes against one table per deployment, the board's `active` and //! `dated` indexes kept beside the rows, next/blocked derived by the same library a local replica -//! uses, and snapshots that are tables plus watermark, without the log. +//! uses, and snapshots that are tables plus watermark, without the log; a type's live items listed +//! by a third index, `typed`, for a ranked hosted `list`. //! //! The pieces, in the order a fold run uses them: [`fold::Folder`] maps each op through the //! platform's reducers and [`write::plan`] turns every change into a [`write::Write`]; @@ -22,27 +23,42 @@ //! | sort key | `sk` | S | `#`, `.adj#`/`.ladj#`/`.tadj##`, `.meta#watermark` | //! | GSI `active` | `nxf_active` / `sk` | S/S | sparse: active tickets and the present dep/parent edges touching one | //! | GSI `dated` | `nxf_dated` / `nxf_dated_at` | S/S | sparse: `#closed` and `#archived`, by `␟` | +//! | GSI `typed` | `nxf_typed` / `sk` | S/S | sparse: `#` on every live item row (not deleted; archived included) | //! //! The `.adj##` entries are plain rows in the stream's partition, on no index. Since //! index revision 3 (6j6v.1c5s) each also carries its edge's `from_id`, `to_id`, `kind`, `lamport`, -//! `site` and a `removed` flag — attributes, not keys: the contract above is unchanged. +//! `site` and a `removed` flag — attributes, not keys. //! -//! Both indexes project ALL attributes, so a lane is one Query and no follow-up reads. The lanes +//! Every index projects ALL attributes, so a lane is one Query and no follow-up reads. +//! +//! **The `typed` index (6j6v.xkr7)** is the one change to this contract since `active` and `dated` +//! (docs/specs/derived-ranking-keys.md §6, owner decision §10.4): one Query lists the live items of +//! a type, the `list` scope a hosted `list` and the derived ranking keys rank over +//! ([`records::list_board`]). Its partition key `nxf_typed` is `#`, the type escaped +//! as a key value is (`%` → `%25`, `#` → `%23`; [`layout::typed_partition`]); its sort key is the +//! table's `sk`. The fold sets it on every item row whose type is set and which is not deleted — +//! closed and archived ones included — and removes it on a delete, written after the row like the +//! other index attributes ([`board`], "`typed`"). The deploying stack adds the GSI FIRST: a fold +//! against a table without it still writes the attribute, which costs nothing and breaks nothing; +//! only the read — a `list` through [`records::list_board`], a `next` whose keys read a +//! `[ranking.list]` rank — needs the index. A type whose partition would pass DynamoDB's 2,048 +//! bytes for an index key is refused at fold time ([`layout::MAX_PARTITION_BYTES`]). The lanes //! need nothing else; the RECORDS a lane is shown with carry joins from other tables, which //! [`records`] reads with a fixed number of requests per active ticket. Billing mode //! is the deployment's call. No local secondary index: without one, DynamoDB sets no limit on the //! size of a stream's partition (the 10 GB item-collection limit applies only to tables with an LSI). //! What does bound a stream is the throughput of ONE partition key: a stream's `active` and `dated` -//! entries each share one index partition — a write-heavy stream is limited to that partition's -//! throughput (1,000 writes per second). +//! entries each share one index partition, its items of one type one `typed` partition — a +//! write-heavy stream is limited to that partition's throughput (1,000 writes per second). //! //! Every row of view table `t` carries its columns as attributes of the same name ([`layout`]); a //! NULL cell is DynamoDB's NULL, an integer a number, text a string. //! //! # Freshness //! -//! A row read (`GetItem`, a Query on the table) is strongly consistent. The two index reads — -//! `board::select` and `board::dated` — are eventually consistent, as every DynamoDB GSI read is: +//! A row read (`GetItem`, a Query on the table) is strongly consistent. The index reads — +//! `board::select`, `board::dated` and the `typed` Query of [`records::list_board`] — are eventually +//! consistent, as every DynamoDB GSI read is: //! right after a fold they may answer the state before it, for as long as the index takes to catch //! up (typically well under a second). The lanes are the local lanes of THAT state. A reader that //! must see its own write reads the ticket's row instead. @@ -68,7 +84,11 @@ //! the selection, at most [`records::READ_CONCURRENCY`] tickets at a time ([`records`]). Folding an //! edge remove reads the add and the remove and rewrites the entry under each end. The graph the //! derived ranking keys read ([`records::with_ranking`]) costs one Query per active ticket and, per -//! inactive neighbour, one `GetItem` plus up to three Queries. No request +//! inactive neighbour, one `GetItem` plus up to three Queries. A hosted `list` +//! ([`records::list_board`]) costs one `typed` Query per listed type and `4n + 1 + p` requests for +//! `n` listed items, as the records read. The `typed` index costs a fold at most one write per item +//! op, and only when the op gains, changes or loses the item's partition — a create, a retype, a +//! delete — in the same write as the item's other index attributes when they move too. No request //! is a Scan: every read names its partition key, the board's through an index. The //! `dynamodb-local` CI job counts the requests of a fold and of a records read, and proves it. //! diff --git a/crates/fold-ddb/src/mem.rs b/crates/fold-ddb/src/mem.rs index 09178aa..155b9b1 100644 --- a/crates/fold-ddb/src/mem.rs +++ b/crates/fold-ddb/src/mem.rs @@ -7,7 +7,7 @@ //! It counts the requests it answers ([`Requests`]), so the cost of a fold — the flag fan-out above //! all — is measured on the same code path the DynamoDB table runs. -use crate::layout::{ACTIVE, DATED, DATED_AT, MAX_ITEM_BYTES, PK, SK}; +use crate::layout::{typed_partition, ACTIVE, DATED, DATED_AT, MAX_ITEM_BYTES, PK, SK, TYPED}; use crate::table::{Dated, Outcome, Row, Table}; use crate::write::{attribute_bytes, beats, Cond, Write}; use nxs_foundation::change::Cell; @@ -176,6 +176,20 @@ impl Table for MemTable { rows.truncate(limit.unwrap_or(usize::MAX)); Ok(rows) } + + async fn query_typed(&self, item_type: &str) -> Result, Infallible> { + self.count(|r| r.index_queries += 1); + let partition = Cell::Text(typed_partition(&self.stream, item_type)); + // The rows are kept by sort key: the index's own order. + Ok(self + .rows + .lock() + .unwrap() + .values() + .filter(|r| r.get(TYPED) == Some(&partition)) + .cloned() + .collect()) + } } /// Run a future that never waits — every [`MemTable`] future — to completion on this thread. diff --git a/crates/fold-ddb/src/records.rs b/crates/fold-ddb/src/records.rs index 4d52e05..6560c95 100644 --- a/crates/fold-ddb/src/records.rs +++ b/crates/fold-ddb/src/records.rs @@ -36,8 +36,17 @@ //! //! One more is read only for a plugin that ranks `next` by derived ranking keys: //! [`with_ranking`] fills the board's [`RankingGraph`] — the edges around the active tickets and -//! their live neighbours — from the adjacency entries (index revision 3, 6j6v.1c5s). Its cost is in -//! its own doc. +//! their live neighbours — from the adjacency entries (index revision 3, 6j6v.1c5s), and the live +//! items of a type whose `[ranking.list]` rank a key reads from the `typed` index (6j6v.xkr7). Its +//! cost is in its own doc. +//! +//! # A hosted `list` +//! +//! [`list_board`] reads what the facade's store-free `list` ranks and projects (6j6v.xkr7): the +//! live, unarchived items of the listed types off the `typed` index, with the joins a `list` +//! record and its filters carry, `next`'s board when a key reads a rank `next` gives, and the +//! graph. The server asks [`read::list_reads`](nexus_flow_facade::read::list_reads) what to read +//! for its query; its cost is in its own doc. //! //! # Two points in time //! @@ -52,27 +61,28 @@ //! # A stream folded by older rules //! //! The entries under a ticket came with index revision 1, the `.cadj#` entries with revision 2 -//! (6j6v.15ed), the self-describing `.adj#` entries with revision 3 (6j6v.1c5s). A stream whose -//! watermark names an older -//! one — or none, folded before revisions were recorded for the index — has no entries, and its -//! records would come back without labels or conversations. [`active_board`] refuses such a stream -//! ([`RecordsError::NotCurrent`]); the fold run refolds it ([`Folder::is_current`]), after which it -//! reads normally. +//! (6j6v.15ed), the self-describing `.adj#` entries and the `typed` index attribute with revision 3 +//! (6j6v.1c5s, 6j6v.xkr7). A stream whose watermark names an older one — or none, folded before +//! revisions were recorded for the index — lacks them, and its records would come back without +//! labels or conversations, its list empty. [`active_board`] and [`list_board`] refuse such a +//! stream ([`RecordsError::NotCurrent`]); the fold run refolds it ([`Folder::is_current`]), after +//! which it reads normally. //! //! [`Folder::is_current`]: crate::fold::Folder::is_current -use crate::board::{int, item_key, Selection, REMOVED}; +use crate::board::{int, is_active, item_key, select, Selection, REMOVED}; use crate::fold::{Folder, INDEX_DOMAIN, INDEX_REVISION}; use crate::layout::{ adjacency_prefix, contributes_adjacency_prefix, label_adjacency_prefix, link_adjacency_prefix, - row_key, + row_key, table_prefix, }; use crate::table::{text, Row, Table}; use futures_util::stream::{self, StreamExt, TryStreamExt}; use nexus_flow_core::graph::Edge; use nexus_flow_core::model::{EdgeKind, ItemRow, LinkWeight}; use nexus_flow_facade::read::{ - ActiveBoard, ActiveTicket, Neighbour, ParentRef, RankingGraph, RankingReads, + ActiveBoard, ActiveTicket, ListBoard, ListItem, ListReads, Neighbour, ParentRef, RankingGraph, + RankingReads, }; use nxs_foundation::change::Cell; use std::collections::{BTreeMap, BTreeSet}; @@ -120,12 +130,7 @@ pub async fn active_board( table: &T, selection: Selection, ) -> Result> { - if let Some(mark) = Folder::watermark(table).await? { - let found = mark.revisions.get(INDEX_DOMAIN).copied(); - if found != Some(INDEX_REVISION) { - return Err(RecordsError::NotCurrent { found }); - } - } + check_current(table).await?; let Selection { items, board, @@ -268,15 +273,19 @@ fn current_parent(id: &str, edges: &BTreeMap) -> Option#` | active ticket | -/// | one `GetItem` of its row | other end of such an edge that is not active | +/// | one Query on the `typed` index | type in `reads.list_types` (6j6v.xkr7) | +/// | one Query on `.adj##` | live, unarchived item of such a type that is not active | +/// | one `GetItem` of its row | other end of such an edge that is not a subject | /// | one Query on `.adj##` (its parent) | such end that is live | -/// | one Query on `.ladj##`, `custom_fields##` | such end, when `reads` asks for labels, custom values | +/// | one Query on `.ladj##`, `custom_fields##` | such end and listed item, when `reads` asks for labels, custom values | /// /// So `n` active tickets with `p` distinct inactive neighbours, `q` of them live, cost `n + q` /// Queries and `p` `GetItem`s — `n + 3q` Queries with labels and custom values — at most /// [`READ_CONCURRENCY`] items at a time, and nothing grows with an item's history: an entry says -/// whether its add is removed. Every edge touching an active ticket is read once, from one end's -/// entries, whatever the plugin's keys — the edge pass the store path runs once per query. +/// whether its add is removed. Every edge touching a subject is read once, from one end's entries, +/// whatever the plugin's keys — the edge pass the store path runs once per query. A `next` key +/// that reads the rank of a type only `[ranking.list]` orders adds, per such type, one index Query +/// and its live items as subjects ([`RankingReads::list_types`]). /// /// [`read::with_ranking`]: nexus_flow_facade::read::with_ranking /// [`read::next_ranking_reads`]: nexus_flow_facade::read::next_ranking_reads @@ -285,28 +294,120 @@ pub async fn with_ranking( mut board: ActiveBoard, reads: &RankingReads, ) -> Result> { - // Every present edge of a kind read that touches an active ticket, once per tag. let ids: Vec = board.tickets.keys().cloned().collect(); - let per_ticket: Vec> = stream::iter(ids) - .map(|id| async move { present_edges(table, &id).await }) + let held: BTreeMap = stream::iter(ids) + .map(|id| async move { + let edges = present_edges(table, &id).await?; + Ok::<_, T::Error>((id, edges)) + }) .buffered(READ_CONCURRENCY) - .try_collect() - .await?; + .try_collect::>() + .await? + .into_iter() + .map(|(id, edges)| { + let parent = board + .tickets + .get(&id) + .and_then(|t| t.item.belongs_to.clone()); + (id, Held { parent, edges }) + }) + .collect(); + board.ranking = Some(ranking_graph(table, reads, held).await?); + Ok(board) +} + +/// A row the board holds itself — an active ticket, a listed item — as the graph reads it: its +/// current parent and the present edges touching it. +struct Held { + parent: Option, + edges: BTreeMap, +} + +/// The live, unarchived items of `item_type` on the `typed` index — the type's `list` scope — +/// asked of the row again, since the index may lag it. In id order. +async fn typed_items(table: &T, item_type: &str) -> Result, T::Error> { + Ok(table + .query_typed(item_type) + .await? + .into_iter() + .filter(|r| { + crate::board::typed(r) == Some(item_type) + && text(r, "archived").is_none() + && text(r, "id").is_some() + }) + .collect()) +} + +/// One live row the graph holds, read in full: its row with its current parent, its labels and +/// custom values when `reads` asks, and its present edges. +async fn neighbour( + table: &T, + id: &str, + row: &Row, + reads: &RankingReads, +) -> Result<(Neighbour, BTreeMap), T::Error> { + let edges = present_edges(table, id).await?; + let mut n = Neighbour::new(item_row(row, current_parent(id, &edges))); + if reads.labels { + n.labels = labels(table, id).await?; + } + if reads.custom { + n.custom = custom(table, id).await?; + } + Ok((n, edges)) +} + +/// The [`RankingGraph`] for `reads` around the rows a board holds itself (`held`, each a subject, +/// with its edges already read) — what [`with_ranking`] and [`list_board`] put on their boards. +async fn ranking_graph( + table: &T, + reads: &RankingReads, + held: BTreeMap, +) -> Result { + // The listed items of `reads.list_types` the board does not hold: subjects too. + let mut listed: BTreeSet = BTreeSet::new(); + let mut scope_rows: Vec<(String, Row)> = Vec::new(); + for ty in &reads.list_types { + for row in typed_items(table, ty).await? { + let id = text(&row, "id").unwrap_or_default().to_string(); + listed.insert(id.clone()); + if !held.contains_key(&id) { + scope_rows.push((id, row)); + } + } + } + let scope: BTreeMap)> = + stream::iter(scope_rows) + .map(|(id, row)| async move { + let read = neighbour(table, &id, &row, reads).await?; + Ok::<_, T::Error>((id, read)) + }) + .buffered(READ_CONCURRENCY) + .try_collect() + .await?; + let subject = |id: &str| held.contains_key(id) || scope.contains_key(id); + + // Every present edge of a kind read that touches a subject, once per tag. let mut touching: BTreeMap = BTreeMap::new(); - for edges in per_ticket { + for edges in held + .values() + .map(|h| &h.edges) + .chain(scope.values().map(|(_, e)| e)) + { touching.extend( edges - .into_iter() - .filter(|(_, e)| reads.kinds.contains(&e.kind)), + .iter() + .filter(|(_, e)| reads.kinds.contains(&e.kind)) + .map(|(tag, e)| (tag.clone(), e.clone())), ); } - // The other ends that are not active: their rows, and for the live ones their current parent - // and the joins asked for. + // The other ends: their rows, and for the live ones their current parent and the joins asked + // for. let others: BTreeSet = touching .values() .flat_map(|e| [&e.from, &e.to]) - .filter(|id| !board.tickets.contains_key(*id)) + .filter(|id| !subject(id)) .cloned() .collect(); let rows: Vec<(String, Option)> = stream::iter(others) @@ -321,16 +422,9 @@ pub async fn with_ranking( .into_iter() .filter_map(|(id, row)| row.map(|r| (id, r))) .filter(|(_, r)| text(r, "deleted") != Some("1")); - let neighbours: BTreeMap = stream::iter(live) + let ends: BTreeMap = stream::iter(live) .map(|(id, row)| async move { - let parent = current_parent(&id, &present_edges(table, &id).await?); - let mut n = Neighbour::new(item_row(&row, parent)); - if reads.labels { - n.labels = labels(table, &id).await?; - } - if reads.custom { - n.custom = custom(table, &id).await?; - } + let (n, _) = neighbour(table, &id, &row, reads).await?; Ok::<_, T::Error>((id, n)) }) .buffered(READ_CONCURRENCY) @@ -339,14 +433,16 @@ pub async fn with_ranking( // The edges between live ends, a `parent` edge only from a child to its current parent. let parent_of = |id: &str| -> Option<&str> { - board - .tickets - .get(id) - .map(|t| &t.item) - .or_else(|| neighbours.get(id).map(|n| &n.item)) - .and_then(|i| i.belongs_to.as_deref()) + match held.get(id) { + Some(h) => h.parent.as_deref(), + None => scope + .get(id) + .map(|(n, _)| n) + .or_else(|| ends.get(id)) + .and_then(|n| n.item.belongs_to.as_deref()), + } }; - let live_end = |id: &str| board.tickets.contains_key(id) || neighbours.contains_key(id); + let live_end = |id: &str| subject(id) || ends.contains_key(id); let edges: BTreeSet = touching .into_values() .filter(|e| live_end(&e.from) && live_end(&e.to)) @@ -358,22 +454,229 @@ pub async fn with_ranking( }) .collect(); // A child whose current parent is elsewhere was read for its parent edge only, which is not in - // the graph: it is no neighbour. - let ends: BTreeSet<&str> = edges + // the graph: it is no neighbour. A listed item always is one. + let edge_ends: BTreeSet<&str> = edges .iter() .flat_map(|e| [e.from.as_str(), e.to.as_str()]) .collect(); - let neighbours = neighbours + let mut neighbours: BTreeMap = ends .into_iter() - .filter(|(id, _)| ends.contains(id.as_str())) + .filter(|(id, _)| edge_ends.contains(id.as_str())) .collect(); + neighbours.extend(scope.into_iter().map(|(id, (n, _))| (id, n))); let mut graph = RankingGraph::new(reads.clone()); graph.edges = edges; graph.neighbours = neighbours; - board.ranking = Some(graph); + graph.listed = listed; + Ok(graph) +} + +/// The [`ListBoard`] for `reads` from the folded tables (6j6v.xkr7) — what [`read::list_board`] +/// reads from a store, item for item — for the facade's store-free `list` +/// ([`read::list_board_page`], [`read::list_board_value`]). A server takes `reads` from +/// [`read::list_reads`] for its query. The stream must be current, as for [`active_board`]: the +/// `typed` index and the self-describing `.adj#` entries came with index revision 3. +/// +/// | read | per | +/// |---------------------------------------------------------|----------------------------------| +/// | one `GetItem` of `.meta#watermark` | board | +/// | one Query on the `typed` index | listed type | +/// | one Query on the partition's `items#` rows | board without a type filter | +/// | one Query each on `.adj#`, `.ladj#`, `custom_fields#`; one `GetItem` of `item_timestamps` | listed item | +/// | one `GetItem` | parent of an open listed item outside the list | +/// | one Query on the `active` index, and per active ticket one Query each on `.ladj#` / `custom_fields#` when the graph reads them | board whose keys read a `next` rank | +/// | [`with_ranking`]'s reads around the listed items and active tickets | board with derived keys | +/// +/// So `n` listed items cost `4n + 1 + p` requests and one Query per listed type, as +/// [`active_board`] costs per active ticket; the `.adj#` Query gives each item's parent, its +/// `contributes_to` targets and its parents for `parent_closed_reason` at once, and the graph reuses +/// it. Archived and deleted items are dropped before any join: `list` returns neither (archived +/// ones are on the index, live as they are, and filtered here as `list` filters them locally). +/// +/// A query without a type filter lists every live item, so it reads the partition's `items#` rows +/// — strongly consistent, every item row of the stream, deleted ones included — instead of an +/// index. The index lists one type; a query that names its types reads only those. +/// +/// [`read::list_board`]: nexus_flow_facade::read::list_board +/// [`read::list_board_page`]: nexus_flow_facade::read::list_board_page +/// [`read::list_board_value`]: nexus_flow_facade::read::list_board_value +/// [`read::list_reads`]: nexus_flow_facade::read::list_reads +pub async fn list_board( + table: &T, + reads: &ListReads, +) -> Result> { + check_current(table).await?; + let rows: Vec = match &reads.types { + Some(types) => { + let mut rows = Vec::new(); + for ty in types { + rows.extend(typed_items(table, ty).await?); + } + rows + } + None => table + .query_prefix(&table_prefix("items")) + .await? + .into_iter() + .filter(|r| { + text(r, "deleted") != Some("1") + && text(r, "archived").is_none() + && text(r, "id").is_some() + }) + .collect(), + }; + let read: Vec<(Row, BTreeMap, ListItem)> = stream::iter(rows) + .map(|row| async move { + let id = text(&row, "id").unwrap_or_default().to_string(); + let edges = present_edges(table, &id).await?; + let (created_at, updated_at) = timestamps(table, &id).await?; + let mut i = ListItem::new(item_row(&row, current_parent(&id, &edges))); + i.labels = labels(table, &id).await?; + i.custom = custom(table, &id).await?; + i.created_at = created_at; + i.updated_at = updated_at; + let targets: BTreeSet = edges + .values() + .filter(|e| e.kind == EdgeKind::ContributesTo && e.from == id) + .map(|e| e.to.clone()) + .collect(); + i.contributes_to = targets.into_iter().collect(); + Ok::<_, T::Error>((row, edges, i)) + }) + .buffered(READ_CONCURRENCY) + .try_collect() + .await?; + + // `parent_closed_reason`: an OPEN item with parents, none of them active, lists them all. A + // parent outside the list is read once, however many children it has. + let parents_of = |id: &str, edges: &BTreeMap| -> BTreeSet { + edges + .values() + .filter(|e| e.kind == EdgeKind::Parent && e.from == id) + .map(|e| e.to.clone()) + .collect() + }; + let listed_rows: BTreeMap<&str, &Row> = read + .iter() + .map(|(row, _, i)| (i.item.id.as_str(), row)) + .collect(); + let outside: BTreeSet = read + .iter() + .filter(|(_, _, i)| i.item.status.as_deref() == Some("open")) + .flat_map(|(_, edges, i)| parents_of(&i.item.id, edges)) + .filter(|p| !listed_rows.contains_key(p.as_str())) + .collect(); + let outside: BTreeMap> = stream::iter(outside) + .map(|pid| async move { + let row = table.get(&item_key(&pid)).await?; + Ok::<_, T::Error>((pid, row)) + }) + .buffered(READ_CONCURRENCY) + .try_collect() + .await?; + let parent_row = |pid: &str| -> Option<&Row> { + listed_rows + .get(pid) + .copied() + .or_else(|| outside.get(pid).and_then(Option::as_ref)) + }; + let parent_closed: Vec)>> = read + .iter() + .map(|(_, edges, i)| { + if i.item.status.as_deref() != Some("open") { + return Vec::new(); + } + let mut closed = Vec::new(); + for pid in parents_of(&i.item.id, edges) { + match parent_row(&pid) { + Some(r) if is_active(r) => return Vec::new(), + Some(r) => closed.push(( + text(r, "id").unwrap_or(&pid).to_string(), + text(r, "closing_comment").map(str::to_string), + )), + None => closed.push((pid, None)), + } + } + closed.sort_by(|a, b| a.0.cmp(&b.0)); + closed + }) + .collect(); + let mut items = Vec::with_capacity(read.len()); + let mut held: BTreeMap = BTreeMap::new(); + for ((_, edges, mut i), closed) in read.into_iter().zip(parent_closed) { + i.parent_closed = closed; + held.insert( + i.item.id.clone(), + Held { + parent: i.item.belongs_to.clone(), + edges, + }, + ); + items.push(i); + } + let mut board = ListBoard::new(reads.types.clone(), items); + + let graph_reads = reads.ranking.clone().unwrap_or_default(); + if reads.next { + let Selection { + items: rows, + board: lanes, + parents, + } = select(table).await?; + let tickets: Vec = stream::iter(rows) + .map(|(id, row)| { + let parent = parents.get(&id).cloned(); + let graph_reads = &graph_reads; + async move { + let mut t = ActiveTicket::new(item_row(&row, parent)); + if graph_reads.labels { + t.labels = labels(table, &id).await?; + } + if graph_reads.custom { + t.custom = custom(table, &id).await?; + } + Ok::<_, T::Error>(t) + } + }) + .buffered(READ_CONCURRENCY) + .try_collect() + .await?; + board.active = Some(ActiveBoard::new(lanes, tickets)); + } + if let Some(r) = &reads.ranking { + if let Some(active) = &board.active { + let missing: Vec<(String, Option)> = active + .tickets + .iter() + .filter(|(id, _)| !held.contains_key(*id)) + .map(|(id, t)| (id.clone(), t.item.belongs_to.clone())) + .collect(); + let more: Vec<(String, Held)> = stream::iter(missing) + .map(|(id, parent)| async move { + let edges = present_edges(table, &id).await?; + Ok::<_, T::Error>((id, Held { parent, edges })) + }) + .buffered(READ_CONCURRENCY) + .try_collect() + .await?; + held.extend(more); + } + board.ranking = Some(ranking_graph(table, r, held).await?); + } Ok(board) } +/// Refuse a stream folded by older index rules than this build reads. +async fn check_current(table: &T) -> Result<(), RecordsError> { + if let Some(mark) = Folder::watermark(table).await? { + let found = mark.revisions.get(INDEX_DOMAIN).copied(); + if found != Some(INDEX_REVISION) { + return Err(RecordsError::NotCurrent { found }); + } + } + Ok(()) +} + /// An `items` row as the store reads it, with the ticket's `belongs_to` (the `present_parent` join). pub(crate) fn item_row(row: &Row, belongs_to: Option) -> ItemRow { let cell = |c: &str| text(row, c).map(str::to_string); diff --git a/crates/fold-ddb/src/table.rs b/crates/fold-ddb/src/table.rs index c382c1e..51cd92b 100644 --- a/crates/fold-ddb/src/table.rs +++ b/crates/fold-ddb/src/table.rs @@ -1,6 +1,6 @@ -//! The five requests a server fold makes of its table, as a trait: [`MemTable`](crate::mem::MemTable) +//! The requests a server fold makes of its table, as a trait: [`MemTable`](crate::mem::MemTable) //! answers them in memory for the tests, `ddb::DynamoDbTable` (feature `dynamodb`) answers them with DynamoDB. -//! None of them is a Scan — every read names its partition, and the two board reads name an index. +//! None of them is a Scan — every read names its partition, and the three index reads name an index. use crate::write::Write; use nxs_foundation::change::Cell; @@ -71,6 +71,13 @@ pub trait Table: Sync { lane: Dated, limit: Option, ) -> impl Future, Self::Error>> + Send; + + /// Every row on the `typed` index under `item_type` (6j6v.xkr7): the live items of that type, + /// archived ones included, in sort-key (id) order. Eventually consistent, as every index read. + fn query_typed( + &self, + item_type: &str, + ) -> impl Future, Self::Error>> + Send; } /// A text cell of `row`, `None` when it is absent or NULL. diff --git a/crates/fold-ddb/tests/common/mod.rs b/crates/fold-ddb/tests/common/mod.rs index eeffa36..8d4274c 100644 --- a/crates/fold-ddb/tests/common/mod.rs +++ b/crates/fold-ddb/tests/common/mod.rs @@ -25,6 +25,10 @@ impl Lcg { } } +/// The item types of a generated board: items are created as one of the first two and retyped +/// among all three, so the `typed` index sees a type gained, lost and changed (6j6v.xkr7). +pub const TYPES: [&str; 3] = ["task", "epic", "note"]; + pub fn id(i: u64) -> String { format!("ab12.{i:04}") } @@ -38,6 +42,11 @@ fn write(s: &mut Store, r: &mut Lcg, n: u64, steps: usize) { step % 24 )); let (a, b) = (id(r.next(n)), id(r.next(n + 1))); + // Now and then a retype (6j6v.xkr7), on the step alone so the random stream — and with it + // every other op of a seed — stays as it was. + if step % 9 == 4 { + s.set_field(&a, "type", Some(TYPES[step % TYPES.len()].into()), "u"); + } match r.next(16) { 0 => s.create_item(&a, "task", "T", "u"), 1 => s.set_field(&a, "status", Some("in_progress".into()), "u"), @@ -149,7 +158,7 @@ pub fn merged(seed: u64) -> Store { let mut a = Store::open_in_memory(1); let mut b = Store::open_in_memory(2); for i in 0..n { - a.create_item(&id(i), "task", "T", "u"); + a.create_item(&id(i), TYPES[(i % 2) as usize], "T", "u"); } b.apply(&a.export()); let steps = 10 + r.next(40) as usize; diff --git a/crates/fold-ddb/tests/dynamodb_local.rs b/crates/fold-ddb/tests/dynamodb_local.rs index 7c49649..d664758 100644 --- a/crates/fold-ddb/tests/dynamodb_local.rs +++ b/crates/fold-ddb/tests/dynamodb_local.rs @@ -1,6 +1,6 @@ //! The DynamoDB adapter against a real DynamoDB Local (6j6v.k7w7): the same fold written into //! DynamoDB and into a [`MemTable`] leaves the same items, the board reads back through the indexes -//! with the same lanes, and every request the client sends is recorded — none is a Scan. +//! with the same lanes and the same `typed` listings (6j6v.xkr7), and every request the client sends is recorded — none is a Scan. //! //! Gated like `nxs-server`'s suite: on the `dynamodb` feature and `NXF_RELAY_DDB_ENDPOINT`, skipped //! locally without one and a hard failure under CI. Each test creates its own table from @@ -16,7 +16,7 @@ use aws_sdk_dynamodb::config::{ use aws_sdk_dynamodb::error::BoxError; use aws_sdk_dynamodb::types::TableStatus; use aws_sdk_dynamodb::Client; -use common::{by_key, delivered, merged, NOW}; +use common::{by_key, delivered, merged, NOW, TYPES}; use nexus_flow_core::model::EdgeKind; use nexus_flow_core::store::Store; use nxs_fold_ddb::ddb::{create_table, DynamoDbTable}; @@ -204,6 +204,20 @@ async fn a_board_folded_into_dynamodb_reads_back_as_from_memory_and_no_request_i assert_eq!(got, want, "{lane:?} limit {limit:?}, seed {seed}"); } } + // The `typed` index (6j6v.xkr7): each type's live items, in id order, through the SDK. + for ty in TYPES { + let ids = |rows: Vec| -> Vec { + rows.iter() + .map(|r| text(r, "id").unwrap().to_string()) + .collect() + }; + let want = ids(mem.query_typed(ty).await.unwrap()); + let got = eventually(&want, || async move { + ids(ddb_ref.query_typed(ty).await.unwrap()) + }) + .await; + assert_eq!(got, want, "typed {ty}, seed {seed}"); + } // Two streams share the table without seeing each other: the second fold's partition // holds only its own rows (checked above), and its reads named its partition. let read = recorder.take(); @@ -309,7 +323,7 @@ async fn a_snapshot_taken_from_dynamodb_starts_another_stream_with_the_same_item // The partition and the index attributes name their stream; everything else is the // same across the two. r.remove("stream_id"); - for attr in ["nxf_active", "nxf_dated"] { + for attr in ["nxf_active", "nxf_dated", "nxf_typed"] { if let Some(nxs_foundation::change::Cell::Text(v)) = r.get_mut(attr) { *v = v .replacen("stream-a", "", 1) diff --git a/crates/fold-ddb/tests/fold_differential.rs b/crates/fold-ddb/tests/fold_differential.rs index ffb765f..824caa8 100644 --- a/crates/fold-ddb/tests/fold_differential.rs +++ b/crates/fold-ddb/tests/fold_differential.rs @@ -6,8 +6,8 @@ //! through the SQLite applier, in the order the replica took it in; and through this crate into a //! [`MemTable`], in a SHUFFLED order with ops delivered twice, as a relay without deduplication //! hands them over. Every row of every view table, every lane of the library and both dated lanes -//! must come out the same, and the incrementally kept indexes must be exactly what a full reindex -//! computes. +//! must come out the same, the `typed` index must list each type's live items (6j6v.xkr7), and the +//! incrementally kept indexes must be exactly what a full reindex computes. use nexus_flow_core::graph; use nexus_flow_core::model::EdgeKind; @@ -15,13 +15,13 @@ use nexus_flow_core::store::Store; use nexus_flow_core::task_reducer::TaskReducer; use nxs_fold_ddb::fold::Folder; use nxs_fold_ddb::mem::{ready, MemTable}; -use nxs_fold_ddb::table::{text, Dated}; +use nxs_fold_ddb::table::{text, Dated, Table}; use nxs_fold_ddb::{board, chunks}; use nxs_foundation::reducer::Reducer; use std::collections::BTreeMap; mod common; -use common::{merged, serve, served_rows, sqlite_rows, NOW}; +use common::{merged, serve, served_rows, sqlite_rows, NOW, TYPES}; fn dated_ids(t: &MemTable, lane: Dated) -> Vec { ready(board::dated(t, lane, None)) @@ -45,6 +45,7 @@ fn the_server_fold_in_any_order_answers_as_the_local_fold() { let folder = Folder::platform().unwrap(); let mut active_seen = 0; let mut chunks_seen = 0; + let mut typed_seen = 0; for seed in 0..120 { let s = merged(seed); let t = serve(&s, seed, &folder); @@ -95,6 +96,24 @@ fn the_server_fold_in_any_order_answers_as_the_local_fold() { "archived lane, {ctx}" ); + // The `typed` index (6j6v.xkr7): under each type exactly its live items, archived ones + // included, in id order. + for ty in TYPES { + let typed: Vec = ready(t.query_typed(ty)) + .unwrap() + .iter() + .map(|r| text(r, "id").unwrap().to_string()) + .collect(); + let want = sql_ids( + &s, + &format!( + "SELECT id FROM items WHERE COALESCE(deleted,'0')<>'1' AND type='{ty}' ORDER BY id" + ), + ); + typed_seen += want.len(); + assert_eq!(typed, want, "typed {ty}, {ctx}"); + } + // The flags kept op by op are exactly what recomputing them from the rows gives. let kept = t.rows(); ready(board::reindex(&t)).unwrap(); @@ -112,6 +131,7 @@ fn the_server_fold_in_any_order_answers_as_the_local_fold() { } assert!(active_seen > 0, "the generated boards have active tickets"); assert!(chunks_seen > 0, "the generated boards supersede chunks"); + assert!(typed_seen > 0, "the generated boards have typed items"); } #[test] diff --git a/crates/fold-ddb/tests/list_differential.rs b/crates/fold-ddb/tests/list_differential.rs new file mode 100644 index 0000000..c44e6fd --- /dev/null +++ b/crates/fold-ddb/tests/list_differential.rs @@ -0,0 +1,441 @@ +//! A server answers a ranked `list` as a replica does (6j6v.xkr7, docs/specs/derived-ranking-keys.md +//! §6 and §9 "differential"), and ranks a `next` whose keys read a `[ranking.list]` rank. +//! +//! Generated boards of four types — goals, projects, actions, persons — written by two replicas +//! apart and then merged: every status, archived and deleted items, retypes, priorities, the goals' +//! `order` custom field, `waiting` labels, and `parent`, `contributes_to`, `mentions` and `dep` +//! edges with removes. Each board is folded into a [`MemTable`] in a shuffled order with +//! duplicates, as the relay hands it over. +//! +//! The plugin ranks with every derived key kind across both sections: persons by a `sum` of +//! mentions, the `list` rank of the goals and the `next` rank of the projects they contribute to; +//! goals by their contributors under `[ranking.list]`; projects under `[ranking.next]` by the +//! `list` rank of their parent goal; actions under `[ranking.next]` by the `list` rank of the +//! persons they mention — the cross-section reads that were refused on the server until the +//! `typed` index — and, in `list`, by the `[ranking.list]` default order with a derived key. +//! +//! Three paths answer each of a set of `list` queries (types, none, container, label, status, +//! sort, limit): +//! +//! 1. the SQL path: `read::list_page` over the store; +//! 2. the Board path: `read::list_board_page` over the store's own [`read::ListBoard`] +//! (`read::list_board` for `read::list_reads`); +//! 3. the fold-ddb path: `read::list_board_page` over the folded tables' (`records::list_board`). +//! +//! All three must give the same rows, order and total, the two boards must be equal item for item +//! and edge for edge, and the JSON must be `Engine::list_query_value`'s. `next` is held to the +//! same three paths. + +use nexus_flow_core::model::{EdgeKind, ItemRow, MergeStrategy}; +use nexus_flow_core::store::Store; +use nexus_flow_facade::plugin::{self, PluginConfig, PluginRegistration}; +use nexus_flow_facade::read::{self, ListQuery, NextFilter, SortKey}; +use nxs_fold_ddb::fold::Folder; +use nxs_fold_ddb::mem::{ready, MemTable}; +use nxs_fold_ddb::{board, records}; +use std::collections::BTreeMap; + +mod common; +use common::{delivered, Lcg, NOW}; + +const ORDERS: &str = r#" +name = "list-differential-fixture" +[description] +en = "x" +de = "y" +[priority] +labels = ["P0", "P1", "P2", "P3", "P4"] +[types] +list = ["goal", "project", "action", "person"] +[vocabulary.status] +open = "open" +in_progress = "in progress" +closed = "closed" +[ranking.next] +types = ["action", "project"] +order = [ { field = "priority", dir = "asc" } ] +[ranking.next.project] +order = [ + { related = "parent", type = "goal", of = "rank", pick = "max", nulls = "first" }, + { related = "contributes_to", type = "goal", of = { field = "order" }, pick = "min", nulls = "last" }, + { count = "children", type = "action", dir = "desc" }, + { field = "priority", dir = "asc" }, +] +[ranking.next.action] +order = [ + { related = "mentions", type = "person", of = "rank", pick = "min", nulls = "last" }, + { related = "parent", type = "project", of = "rank", nulls = "last" }, + { related = "dep", of = { field = "status", precedence = ["closed", "in_progress", "open"] }, pick = "min" }, + { field = "priority", dir = "asc" }, +] +[ranking.list] +types = ["person", "goal", "project"] +order = [ + { count = "mentioned_by", dir = "desc" }, + { field = "priority", dir = "asc" }, +] +[ranking.list.goal] +order = [ + { count = "contributors", type = "project", dir = "desc" }, + { field = "priority", dir = "asc" }, +] +[ranking.list.person] +order = [ + { sum = [ { count = "mentioned_by", type = "project" }, + { count = "mentioned_by", type = "action", label = "waiting", status = ["open", "in_progress"] } ], dir = "desc" }, + { related = "contributes_to", type = "goal", of = "rank", pick = "min", nulls = "last" }, + { related = "contributes_to", type = "project", of = "rank", pick = "min", nulls = "last" }, + { count = "dependents", status = ["closed"], dir = "desc" }, + { field = "priority", dir = "asc" }, +] +[presentation.list] +columns = ["id"] +[fields.order] +type = "text" +on = ["goal"] +"#; + +inventory::submit!(PluginRegistration { + name: "list-differential-fixture", + toml: ORDERS, + order: 141, +}); + +fn cfg() -> PluginConfig { + plugin::load("list-differential-fixture").expect("the fixture is a valid plugin") +} + +const TYPES: [&str; 4] = ["goal", "project", "action", "person"]; + +/// The items of a generated board: three goals, four projects, six actions, five persons. +fn items() -> Vec<(String, &'static str)> { + let mut out = Vec::new(); + for (ty, letter, n) in [ + ("goal", 'g', 3), + ("project", 'p', 4), + ("action", 'a', 6), + ("person", 'u', 5), + ] { + for i in 0..n { + out.push((format!("ab12.{letter}{i:03}"), ty)); + } + } + out +} + +/// One replica's share of the writing. +fn write(s: &mut Store, r: &mut Lcg, added: &mut Vec<(String, String, EdgeKind)>, steps: usize) { + let all = items(); + let pick = |r: &mut Lcg| all[r.next(all.len() as u64) as usize].0.clone(); + for step in 0..steps { + s.set_wall_clock(&format!( + "2026-10-0{}T{:02}:00:00Z", + 1 + r.next(7), + step % 24 + )); + let (x, y) = (pick(r), pick(r)); + match r.next(18) { + 0 => s.set_field(&x, "status", Some("in_progress".into()), "u"), + 1 | 2 => s.set_field(&x, "status", Some("closed".into()), "u"), + 3 => s.set_field(&x, "status", Some("open".into()), "u"), + 4 => s.set_field( + &x, + "archived", + Some(format!("2026-10-0{}", 1 + r.next(7))), + "u", + ), + 5 => s.set_field(&x, "archived", None, "u"), + 6 if r.next(3) == 0 => s.delete_item(&x, "u"), + 7 => s.set_field(&x, "priority", Some(r.next(5).to_string()), "u"), + 8..=12 => { + let kind = match r.next(10) { + 0..=2 => EdgeKind::Parent, + 3..=5 => EdgeKind::ContributesTo, + 6..=8 => EdgeKind::Mentions, + _ => EdgeKind::Dep, + }; + s.add_edge(&x, &y, kind, "u"); + added.push((x, y, kind)); + } + 13 if !added.is_empty() => { + let (from, to, kind) = added.swap_remove(r.next(added.len() as u64) as usize); + s.remove_edge(&from, &to, kind, "u"); + } + 14 => { + if r.next(3) == 0 { + s.remove_label(&x, "waiting", "u"); + } else { + s.add_label(&x, "waiting", "u"); + } + } + 15 if r.next(3) == 0 => { + s.set_field(&x, "type", Some(TYPES[r.next(4) as usize].into()), "u") + } + _ => s.set_custom_field_merge( + &x, + "order", + Some(r.next(6).to_string()), + "u", + MergeStrategy::Lww, + ), + } + } +} + +/// A merged board: two replicas that wrote apart, then exchanged everything. +fn gen_board(seed: u64) -> Store { + let mut r = Lcg(seed ^ 0x115d); + let mut a = Store::open_in_memory(1); + for (id, ty) in items() { + a.create_item(&id, ty, "T", "u"); + a.set_field(&id, "priority", Some(r.next(5).to_string()), "u"); + if ty == "goal" && r.next(4) != 0 { + a.set_custom_field_merge( + &id, + "order", + Some(r.next(6).to_string()), + "u", + MergeStrategy::Lww, + ); + } + } + let mut b = Store::open_in_memory(2); + b.apply(&a.export()); + let steps = 30 + r.next(40) as usize; + write(&mut a, &mut r, &mut Vec::new(), steps); + write(&mut b, &mut r, &mut Vec::new(), steps); + let from_b = b.export(); + a.apply(&from_b); + a +} + +fn ids(items: &[ItemRow]) -> Vec { + items.iter().map(|i| i.id.clone()).collect() +} + +/// The `list` queries each board is asked: a type, several, none; a container, a label, a status; +/// the default order; a cut. +fn queries(container: &str) -> Vec { + let ranked = || ListQuery::new().sort(SortKey::Rank).now(NOW); + let types = |tys: &[&str]| tys.iter().fold(NextFilter::new(), |f, t| f.with_type(*t)); + vec![ + ranked().filter(types(&["person"])), + ranked().filter(types(&["goal"])), + ranked().filter(types(&["project", "person"])), + ranked(), + ranked().filter(types(&["action"]).with_container(container)), + ranked().filter(NextFilter::new().with_label("waiting")), + ranked().status("closed").filter(types(&["person"])), + ListQuery::new().filter(types(&["person"])), + ranked().filter(types(&["person"])).limit(3), + ] +} + +/// What the generated boards held, so the comparison is known to have compared something worth +/// comparing. +#[derive(Default)] +struct Seen(BTreeMap<&'static str, usize>); + +impl Seen { + fn add(&mut self, what: &'static str, n: usize) { + *self.0.entry(what).or_default() += n; + } +} + +#[test] +fn the_server_answers_a_ranked_list_as_the_replica_does() { + let folder = Folder::platform().unwrap(); + let cfg = cfg(); + let mut seen = Seen::default(); + for seed in 0..40 { + let s = gen_board(seed); + let ctx = format!("seed {seed}"); + let t = MemTable::new("stream-1"); + ready(folder.fold_batch(&t, &delivered(&s, seed))).unwrap(); + + for (n, q) in queries(&format!("ab12.p{:03}", seed % 4)) + .iter() + .enumerate() + { + let ctx = format!("{ctx}, query {n}"); + let reads = read::list_reads(&cfg, q); + let local = read::list_board(&s, &reads).unwrap(); + let server = ready(records::list_board(&t, &reads)).unwrap(); + assert_eq!(server.items, local.items, "listed items, {ctx}"); + assert_eq!(server.ranking, local.ranking, "the ranking graph, {ctx}"); + assert_eq!( + server.active.as_ref().map(|a| &a.tickets), + local.active.as_ref().map(|a| &a.tickets), + "next's board, {ctx}" + ); + + // 1 = 2 = 3: the SQL path, the Board path, the fold-ddb path. + let want = read::list_page(&cfg, &s, q).unwrap(); + let board_path = read::list_board_page(&cfg, &local, q).unwrap(); + let served = read::list_board_page(&cfg, &server, q).unwrap(); + assert_eq!( + ids(&board_path.items), + ids(&want.items), + "SQL vs Board, {ctx}" + ); + assert_eq!( + ids(&served.items), + ids(&want.items), + "SQL vs fold-ddb, {ctx}" + ); + assert_eq!(served.total, want.total, "total, {ctx}"); + let records = read::list_to_value_with_custom(&cfg, &s, &want.items).unwrap(); + assert_eq!( + read::list_board_value(&cfg, &server, q).unwrap(), + read::next_page_envelope(records, &want, false), + "list JSON, {ctx}" + ); + + seen.add("listed rows", want.items.len()); + if let Some(graph) = &server.ranking { + seen.add("graph listed items", graph.listed.len()); + seen.add("graph edges", graph.edges.len()); + } + seen.add( + "boards with next's board", + usize::from(server.active.is_some()), + ); + seen.add( + "listed items with a closed-parent reason", + server + .items + .values() + .filter(|i| !i.parent_closed.is_empty()) + .count(), + ); + seen.add( + "closed persons listed", + want.items + .iter() + .filter(|i| { + i.item_type.as_deref() == Some("person") + && i.status.as_deref() == Some("closed") + }) + .count(), + ); + } + + // `next` whose keys read the `list` rank of persons and goals (spec §12 point 1). + let reads = read::next_ranking_reads(&cfg) + .unwrap() + .expect("derived next keys"); + let selection = ready(board::select(&t)).unwrap(); + let plain = ready(records::active_board(&t, selection)).unwrap(); + let plain = ready(records::with_contributes_to(&t, plain)).unwrap(); + let server = ready(records::with_ranking(&t, plain, &reads)).unwrap(); + let local = read::with_ranking(&s, read::active_board(&s).unwrap(), &reads).unwrap(); + assert_eq!(server.ranking, local.ranking, "next's graph, {ctx}"); + let want = read::next(&cfg, &s, NOW, None).unwrap(); + assert_eq!( + ids(&read::next_active(&cfg, &local, NOW).unwrap()), + ids(&want), + "next: SQL vs Board, {ctx}" + ); + assert_eq!( + ids(&read::next_active(&cfg, &server, NOW).unwrap()), + ids(&want), + "next: SQL vs fold-ddb, {ctx}" + ); + seen.add("next rows", want.len()); + seen.add( + "archived items", + s.list_items() + .unwrap() + .iter() + .filter(|i| i.archived.is_some() && i.deleted.as_deref() != Some("1")) + .count(), + ); + } + for what in [ + "listed rows", + "graph listed items", + "graph edges", + "boards with next's board", + "listed items with a closed-parent reason", + "closed persons listed", + "next rows", + "archived items", + ] { + assert!( + seen.0.get(what).copied().unwrap_or(0) > 0, + "the generated boards have {what}: {:?}", + seen.0 + ); + } +} + +#[test] +fn a_stream_not_folded_by_this_index_revision_is_refused_for_a_list() { + // The `typed` index and the self-describing adjacency entries come with index revision 3: a + // stream folded before has neither, and its list would come back empty — refused instead. + let folder = Folder::platform().unwrap(); + let cfg = cfg(); + let s = gen_board(5); + let t = MemTable::new("stream-1"); + ready(folder.fold_batch(&t, &delivered(&s, 5))).unwrap(); + use nxs_fold_ddb::fold::INDEX_DOMAIN; + use nxs_fold_ddb::layout::WATERMARK_KEY; + use nxs_fold_ddb::table::Table; + use nxs_fold_ddb::write::{Cond, Write}; + use nxs_foundation::change::Cell; + ready(t.write(&Write { + sk: WATERMARK_KEY.into(), + set: vec![(format!("revision_{INDEX_DOMAIN}"), Cell::Int(2))], + remove: Vec::new(), + condition: Some(Cond::RowPresent), + })) + .unwrap(); + let q = &queries("ab12.p000")[0]; + match ready(records::list_board(&t, &read::list_reads(&cfg, q))) { + Err(records::RecordsError::NotCurrent { found: Some(2) }) => {} + other => panic!("a revision-2 stream must be refused, got {other:?}"), + } +} + +#[test] +fn reading_a_typed_list_costs_one_index_query_and_four_requests_per_item() { + // §9 "bounded cost": the persons of a board are one Query on the `typed` index, and each costs + // its adjacency, labels, custom values and timestamps — whatever else the board holds, which + // this list reads nothing of. A type filter reads only its own type. + let folder = Folder::platform().unwrap(); + let cfg = cfg(); + let mut s = Store::open_in_memory(1); + let n = 8u64; + for i in 0..n { + s.create_item(&format!("ab12.u{i:03}"), "person", "P", "u"); + } + for i in 0..30 { + s.create_item(&format!("ab12.a{i:03}"), "action", "A", "u"); + } + s.create_item("ab12.uzzz", "person", "Archived", "u"); + s.set_field("ab12.uzzz", "archived", Some("2026-10-01".into()), "u"); + let t = MemTable::new("stream-1"); + ready(folder.fold_batch(&t, &delivered(&s, 1))).unwrap(); + let q = ListQuery::new().filter(NextFilter::new().with_type("person")); + let reads = read::list_reads(&cfg, &q); + assert_eq!(reads.ranking, None); + assert!(!reads.next); + let before = t.requests(); + let server = ready(records::list_board(&t, &reads)).unwrap(); + let after = t.requests(); + assert_eq!(after.index_queries - before.index_queries, 1); + assert_eq!(after.queries - before.queries, 3 * n); + // The watermark, and each person's timestamps; no parents. + assert_eq!(after.gets - before.gets, 1 + n); + assert_eq!(after.writes, before.writes); + assert_eq!( + server.items.len(), + n as usize, + "the archived person is not listed" + ); + let page = read::list_board_page(&cfg, &server, &q).unwrap(); + assert_eq!( + ids(&page.items), + ids(&read::list_page(&cfg, &s, &q).unwrap().items) + ); +} diff --git a/crates/fold-ddb/tests/robustness.rs b/crates/fold-ddb/tests/robustness.rs index 414a3f3..815e3b1 100644 --- a/crates/fold-ddb/tests/robustness.rs +++ b/crates/fold-ddb/tests/robustness.rs @@ -1,6 +1,7 @@ //! What a server fold does when things go wrong (review of PR #24): a run that dies at any write //! and folds its batch again, an op the table cannot hold, a clear that dies half way, a release -//! with other fold revisions, a lane cut by a limit, and a snapshot that does not fit. +//! with other fold revisions, a lane cut by a limit, a snapshot that does not fit — and the `typed` +//! index kept through retypes, archives and deletes (6j6v.xkr7). mod common; @@ -94,6 +95,10 @@ impl Table for Failing<'_> { async fn query_dated(&self, lane: Dated, limit: Option) -> Result, Injected> { Ok(ok(self.inner.query_dated(lane, limit).await)) } + + async fn query_typed(&self, item_type: &str) -> Result, Injected> { + Ok(ok(self.inner.query_typed(item_type).await)) + } } /// Fold `first` cleanly, then `second` with a run that dies at each write in turn and folds @@ -214,6 +219,107 @@ fn an_op_the_table_cannot_hold_is_refused_and_the_stream_folds_on() { assert_eq!(lanes.ready, vec!["ab12.0001", "ab12.0002"]); } +#[test] +fn a_type_too_long_for_the_typed_index_is_refused_and_the_item_folds_on_without_it() { + // 6j6v.xkr7: the `typed` index partition carries the type; DynamoDB refuses an index key past + // 2,048 bytes, and a refused write would fail every run at the same op. So the op that sets + // such a type is refused before it writes, as an op with a key too long is, and the item keeps + // the type it had. + let folder = Folder::platform().unwrap(); + let mut s = Store::open_in_memory(1); + s.create_item("ab12.0001", "task", "T", "u"); + s.set_field("ab12.0001", "type", Some("t".repeat(2100)), "u"); + s.create_item("ab12.0002", &"n".repeat(2030), "Just fits", "u"); + let t = MemTable::new("stream-1"); + let refused = ready(folder.fold_batch(&t, &numbered(&s, 0))).unwrap(); + let reasons: Vec<&str> = refused.iter().map(|r| r.reason.as_str()).collect(); + assert_eq!(refused.len(), 1, "{reasons:#?}"); + assert!( + reasons[0].contains("`typed` index partition"), + "{reasons:#?}" + ); + let ids = |ty: &str| -> Vec { + ready(t.query_typed(ty)) + .unwrap() + .iter() + .map(|r| text(r, "id").unwrap().to_string()) + .collect() + }; + assert_eq!(ids("task"), ["ab12.0001"]); + assert_eq!(ids(&"n".repeat(2030)), ["ab12.0002"]); +} + +#[test] +fn retyping_archiving_and_deleting_move_an_item_on_the_typed_index() { + // 6j6v.xkr7: an item is listed under its type while it is live — archived or not — moves with a + // retype, and leaves when it is deleted. A type with `#` in it gets its own partition. + let folder = Folder::platform().unwrap(); + let mut s = Store::open_in_memory(1); + s.create_item("ab12.0001", "person", "Ada", "u"); + s.create_item("ab12.0002", "person", "Bob", "u"); + s.create_item("ab12.0003", "a#b", "Odd", "u"); + let t = MemTable::new("stream-1"); + let mut seen = 0; + let mut step = |s: &Store| { + let ops = numbered(s, seen); + seen += ops.len(); + ready(folder.fold_batch(&t, &ops)).unwrap(); + ["person", "project", "a#b", "a"].map(|ty| { + ready(t.query_typed(ty)) + .unwrap() + .iter() + .map(|r| text(r, "id").unwrap().to_string()) + .collect::>() + }) + }; + let v = |ids: &[&str]| ids.iter().map(|s| s.to_string()).collect::>(); + let none = v(&[]); + + assert_eq!( + step(&s), + [ + v(&["ab12.0001", "ab12.0002"]), + none.clone(), + v(&["ab12.0003"]), + none.clone() + ] + ); + s.set_field("ab12.0002", "archived", Some("2026-10-08".into()), "u"); + s.set_field("ab12.0001", "status", Some("closed".into()), "u"); + assert_eq!( + step(&s), + [ + v(&["ab12.0001", "ab12.0002"]), + none.clone(), + v(&["ab12.0003"]), + none.clone() + ], + "closed and archived items stay listed" + ); + s.set_field("ab12.0001", "type", Some("project".into()), "u"); + assert_eq!( + step(&s), + [ + v(&["ab12.0002"]), + v(&["ab12.0001"]), + v(&["ab12.0003"]), + none.clone() + ] + ); + s.delete_item("ab12.0002", "u"); + s.delete_item("ab12.0003", "u"); + assert_eq!( + step(&s), + [none.clone(), v(&["ab12.0001"]), none.clone(), none.clone()] + ); + // The attribute is on the item rows only, never on an adjacency entry or another row. + assert!(t + .rows() + .iter() + .filter(|(_, r)| r.contains_key("nxf_typed")) + .all(|(sk, _)| sk.starts_with("items#"))); +} + #[test] fn a_clear_that_dies_keeps_the_watermark_and_a_release_with_other_rules_refolds_from_zero() { let folder = Folder::platform().unwrap(); diff --git a/docs/specs/E4-sync-protocol-server.md b/docs/specs/E4-sync-protocol-server.md index a63c679..9dc3d5c 100644 --- a/docs/specs/E4-sync-protocol-server.md +++ b/docs/specs/E4-sync-protocol-server.md @@ -278,6 +278,13 @@ relay — does it with `nxs-fold-ddb`: the reducers a local replica registers, t - **GSI `dated`** (`nxf_dated` / `nxf_dated_at`): sparse — `#closed` by `closed_at`, `#archived` by `archived`, newest first; the sort key is `U+001F`, so tickets of one instant keep one order and a `limit` cuts the lane at the same place every time. +- **GSI `typed`** (`nxf_typed` / `sk`, both S, projecting all; added 2026-10-11, `6j6v.xkr7`): + sparse — `#` (the type escaped as a key value: `%` → `%25`, `#` → `%23`) on every + live item row, closed and archived ones included, removed on a delete. One Query lists a type's + live items, the `list` scope a ranked hosted `list` and the derived ranking keys rank over + (`records::list_board`, `docs/specs/derived-ranking-keys.md` §6). The deploying stack adds the + GSI first: a fold against a table without it still writes the attribute harmlessly, and only + the read needs the index. - **The active tickets' records** (added 2026-10-10, `6j6v.t1ym`). The LANES need only the `active` Query above. The RECORDS a lane is shown with need more: a lane's JSON carries joins kept in other tables — labels, custom values, timestamps, bearing thread links, the parent's diff --git a/docs/specs/derived-ranking-keys.md b/docs/specs/derived-ranking-keys.md index cf37db7..3682d4f 100644 --- a/docs/specs/derived-ranking-keys.md +++ b/docs/specs/derived-ranking-keys.md @@ -3,8 +3,9 @@ **Status: decided by the owner on 2026-10-09** (§10). The cuts that build it are listed in §11. **Built locally (x0wf, §12):** the three key kinds, their load-time rules and the type chain over the SQL path, in `next` and `list`. **Built on the server for `next` (1c5s, §6):** the same keys -over the hosted board, with the rule the owner decided on 2026-10-11 (§10.5). The `typed` index and -a ranked hosted `list` (xkr7) are open. +over the hosted board, with the rule the owner decided on 2026-10-11 (§10.5). **Built on the server +for `list` (xkr7, §6):** the `typed` index, a ranked hosted `list`, and a hosted `next` whose keys +read a `[ranking.list]` rank. Fourth cut of `6j6v.kfs7` (cut with the owner on 2026-10-09). The three cuts before it add the plumbing this one ranks with: @@ -164,14 +165,18 @@ derived key pays nothing. adjacency; per neighbour that is not active one `GetItem` of its row and, when it is live, one Query on its own adjacency for its current parent, plus one Query each for its labels and custom values when a key reads them. -- **Refused on the server until xkr7:** a `[ranking.next]` order that reads — directly or down its - type chain — the rank of a type only `[ranking.list]` orders (§12, first point). That rank is - computed over the type's live items, which the active board does not hold. The store path ranks - such a plugin. Every other derived `next` key ranks on the server. +- **A `[ranking.list]` rank in a `next` key** (refused on the server in 1c5s, ranked since xkr7): a + `[ranking.next]` order that reads — directly or down its type chain — the rank of a type only + `[ranking.list]` orders (§12, first point). That rank is computed over the type's live, + unarchived items, which the active board does not hold; `read::next_ranking_reads` names such a + type in `RankingReads::list_types`, and the server reads its items off the `typed` index into the + graph (`RankingGraph::listed`), with the edges around them. - **What xkr7 builds on.** The self-describing adjacency entries are written for EVERY edge between - ANY items, so xkr7's "widened adjacency" needs no new entries for the edges: a `list`-ranked item - reads its edges with the same Query. What xkr7 adds is the `typed` index (the `list` scope of a - type), the read that ranks a type's live items from it, and lifting the refusal above. + ANY items, so xkr7's "widened adjacency" needed no new entries for the edges: a `list`-ranked item + reads its edges with the same Query (verified in xkr7: its `.adj#` entries give its current + parent, its `contributes_to` targets, its parents for `parent_closed_reason` and every edge a key + follows or counts). What xkr7 adds is the `typed` index (the `list` scope of a type), the read + that ranks a type's live items from it, and lifting the refusal above. **`list` needs a second index (decided §10.4).** Objects that `list` ranks, such as a person, are long-lived and have no "active" status. If they did, they would rank through `next`, and `list` @@ -191,8 +196,7 @@ gains a second sparse index, keyed by type: (`.adj#`) were always written for every edge add, under both ends; only the `active` INDEX flag is limited to `dep`/`parent` edges with an active end. Since index revision 3 the entries say what their edge is and whether it is removed (`next` above), so a `list`-ranked item reads its edges - with one Query on its own prefix, at no extra cost per edge write. What stays open for xkr7 is - the read of a type's live items through `typed` and their neighbours' rows. + with one Query on its own prefix, at no extra cost per edge write. - **Archived items.** They are in the index, because they are live. `list` filters them at read, as it does locally. - **It changes the table contract.** The deploying stack (the web package's CDK) must add the GSI @@ -200,6 +204,32 @@ gains a second sparse index, keyed by type: still writes the attribute, which costs nothing. Only the `list` read needs the index. The rollout order is therefore: the stack first, then the read. +**`list` as built (xkr7).** +- **The key.** `nxf_typed` is `#` with the type escaped as a key value is (`%` → + `%25`, `#` → `%23`), so `a` + `b#c` and `a#b` + `c` cannot share a partition; the sort key is the + table's `sk`, so a Query lists a type's items in id order. An item without a type is on no + partition. A type whose partition would pass the 2,048 bytes DynamoDB allows an index key is + refused at fold time, as an op with a sort key too long is. +- **Index revision.** The attribute ships in revision 3 with the self-describing adjacency entries + (1c5s and xkr7 reach the same beta), so revision 3 still makes every older stream not current + and one refold sets both; no revision 4. +- **Where it lives.** The facade's `read::list_reads(cfg, query)` names what a hosted `list` reads + (`ListReads`: the listed types — the query's type filter, else every type —, whether `next`'s + board is needed, and the graph); `nxs_fold_ddb::records::list_board` reads it into a `ListBoard` + (the listed items with the joins a `list` record and its filters carry, `next`'s board, the + `RankingGraph`); `read::list_board_page` and `read::list_board_value` filter, rank and project it + by the code `list_page` runs, and `read::list_board` reads the same board from a store, the + parity oracle. +- **Without a type filter** the server lists every live item: it reads the partition's `items#` + rows (strongly consistent, one Query, no index), since the index lists one type at a time. +- **Paging.** As for the hosted `next`: the store path's page cache is machine-local and a + server's paging is an open decision (15ed), so the store-free `list` is not paged — `limit` cuts + it with the total disclosed, `paginate` and a token are refused with `validation`, and a server + cuts the list itself. +- **`now`.** A `list` key that reads a rank `next` gives takes `next`'s candidates at the query's + `now`, as locally (§12); without one the read answers `validation` where a ranked row needs it, + and `next`'s board is not read for it. + A differential test holds the SQL path and the server path to the same order on generated boards, for `next` and for `list`, as for the lanes today. @@ -259,7 +289,10 @@ the plugin declares (`plugin-custom-fields.md`), as decided in §10.2. - **Differential:** SQL path against the server path on generated boards with derived keys, as `fold_differential` does for the lanes. Built for `next` in 1c5s (`crates/fold-ddb/tests/derived_differential.rs`): the SQL path, the store-free path over the - store's own board and graph, and the same over the folded tables give one order. + store's own board and graph, and the same over the folded tables give one order. Built for + `list` in xkr7 (`crates/fold-ddb/tests/list_differential.rs`): the same three paths on + generated boards of four types with retypes, every key kind and cross-section ranks both ways, + over a set of queries (types, none, container, label, status, sort, limit), and the JSON. - **Determinism:** shuffled edge order gives the same result, and ties fall to the id. - **Bounded cost:** a ratio guard in the style of `xn8s`. A query with derived keys runs the edge pass once, not once per candidate. @@ -288,8 +321,8 @@ the plugin declares (`plugin-custom-fields.md`), as decided in §10.2. adjacency entries (index revision 3) and the server reads the neighbours that are not active. Narrowing the local rule to active neighbours was declined, as was refusing on the server every key whose value can depend on a neighbour that is not active (it would refuse §8's first order). - A `[ranking.next]` order that reads the rank of a type only `[ranking.list]` orders is refused on - the server until the `typed` index (xkr7) gives it that type's scope. + A `[ranking.next]` order that reads the rank of a type only `[ranking.list]` orders was refused on + the server until the `typed` index (xkr7) gave it that type's scope; it ranks there since. ## 11. Cuts @@ -299,8 +332,11 @@ the plugin declares (`plugin-custom-fields.md`), as decided in §10.2. lanes' board, the self-describing adjacency entries (index revision 3), and the differential test (SQL path = Board path = fold-ddb path on generated boards with closed, archived and deleted neighbours, mentions and inbound `contributes_to`). -- **The `typed` index and a ranked hosted `list`.** The GSI, the widened adjacency, the read, and - the table-contract note for the deploying stack, coordinated with app-foundations. +- **The `typed` index and a ranked hosted `list`** (xkr7, built). The GSI, the read + (`ListBoard`, `records::list_board`, `read::list_board_page`), the `[ranking.list]` rank in a + hosted `next`, the differential test, and the table-contract note for the deploying stack + (`nxs-fold-ddb` crate doc, E4 §5.1), coordinated with app-foundations. The widened adjacency + turned out to be 1c5s's self-describing entries; nothing more was needed. - **Shared with `6j6v.q29p`.** The reverse mention lookup and `count = "mentioned_by"` use one implementation. @@ -334,7 +370,9 @@ reading of the text, so a later decision can loosen it without breaking a declar takes them. - **`nulls` on a `count` or `sum`** is refused: a count is never null (§3.2). - **The store-free lanes** (`read::next_active`) rank a `[ranking.next]` with derived keys from - the board's `RankingGraph` since 1c5s (§6), with the live-neighbour rule above; they refuse only - an order that reads the rank of a type only `[ranking.list]` orders, until xkr7. + the board's `RankingGraph` since 1c5s (§6), with the live-neighbour rule above; since xkr7 an + order that reads the rank of a type only `[ranking.list]` orders too, from the graph's listed + items. The store-free `list` (`read::list_board_page`, xkr7) ranks a `[ranking.list]` the same + way. - **TOML.** An inline table may not span lines outside its arrays (TOML 1.0), so a `sum`'s `dir` stands on the line its array closes on. §8 above breaks that line for reading only. From 9e8e68e627e00f6730896e46a428b370b02a6a01 Mon Sep 17 00:00:00 2001 From: Carsten Koch Date: Sun, 11 Oct 2026 21:43:34 +0200 Subject: [PATCH 3/3] Review mitigation for the 1c5s/xkr7 batch (nxf 6j6v.1c5s, xkr7) Mitigates the independent review of PR #55 (.nxf/review/2026-10-11T18-13-12Z.md): - Bounded hosted read cost: an unranked list keeps and cuts on the item rows before any join (ListReads::select, read::list_select, ListBoard::cut); point reads go by BatchGetItem (Table::get_many, 100 keys a request, UnprocessedKeys retried with backoff); request-count guard tests for the untyped list and a ranked list with next + graph. A ranked list still reads its types' whole list scope (spec section 12). - Closed the refold window: Folder::clear(table, until) rewrites the watermark first as a sentinel without revisions; readers answer NotCurrent, then Refolding until the refold reaches `until`. Snapshot import writes the same sentinel first. - records::with_ranking checks the index revision; the revision-2 test reads the stripped entries through it. - An item op whose row would pass 400 KB with its index attributes is refused before it writes; refresh_item's Outcome is surfaced. - present_edges drops entries that do not name their item; typed_items doc; redundant graph_for removed; targeted seen counters; typed index re-create and type None; typed_partition escaping; DynamoDB Local coverage of list_board / with_ranking including a multi-page hub. Co-Authored-By: Claude Opus 5.5 Claude-Session: https://claude.ai/code/session_016q5uMUkZJQW4fckJoA3FME --- changes/1c5s-server-derived-keys.md | 4 +- changes/xkr7-typed-index.md | 4 +- crates/facade/src/read.rs | 152 ++++++++- crates/facade/src/read/derived.rs | 35 ++- crates/facade/tests/derived_ranking.rs | 14 + crates/fold-ddb/Cargo.toml | 2 +- crates/fold-ddb/src/board.rs | 83 ++++- crates/fold-ddb/src/ddb.rs | 75 ++++- crates/fold-ddb/src/fold.rs | 173 +++++++++-- crates/fold-ddb/src/lib.rs | 19 +- crates/fold-ddb/src/mem.rs | 15 +- crates/fold-ddb/src/records.rs | 290 +++++++++++++----- crates/fold-ddb/src/snapshot.rs | 32 +- crates/fold-ddb/src/table.rs | 13 + crates/fold-ddb/tests/common/mod.rs | 6 + crates/fold-ddb/tests/derived_differential.rs | 63 +++- crates/fold-ddb/tests/dynamodb_local.rs | 177 ++++++++++- crates/fold-ddb/tests/fold_differential.rs | 19 +- crates/fold-ddb/tests/list_differential.rs | 258 +++++++++++++++- crates/fold-ddb/tests/records_differential.rs | 6 +- crates/fold-ddb/tests/robustness.rs | 255 ++++++++++++++- docs/specs/E4-sync-protocol-server.md | 39 ++- docs/specs/derived-ranking-keys.md | 37 ++- 23 files changed, 1567 insertions(+), 204 deletions(-) diff --git a/changes/1c5s-server-derived-keys.md b/changes/1c5s-server-derived-keys.md index 58aed9e..cd0fa3a 100644 --- a/changes/1c5s-server-derived-keys.md +++ b/changes/1c5s-server-derived-keys.md @@ -3,6 +3,6 @@ type: added facade: changed --- [en] -A hosted board now ranks `next` by derived ranking keys (`related`, `count`, `sum`) exactly as a local replica does. The store-free `next` (`read::next_active`, `next_active_value`, `next_active_filtered`) ranks a plugin with such keys from a new `ActiveBoard::ranking` graph: the edges around the active tickets and the live neighbours they reach, closed and archived ones included, deleted ones not. A server fills it with `nxs_fold_ddb::records::with_ranking` for what `read::next_ranking_reads(cfg)` names; `read::with_ranking` reads the same graph from a store. Without the graph, `next_active` still refuses such a plugin. Everything is additive: `ActiveBoard` gains a field (it is `#[non_exhaustive]`), and `RankingReads`, `RankingGraph`, `Neighbour`, `next_ranking_reads` and `with_ranking` are new. For a server's operators: the fold's index revision goes from 2 to 3, so every hosted stream is not current and is folded again on its next run; until then its records read answers `NotCurrent`. The `.adj#` adjacency entries gain attributes; the one table-contract change of this release, the `typed` index, is described with the hosted `list`. +A hosted board now ranks `next` by derived ranking keys (`related`, `count`, `sum`) exactly as a local replica does. The store-free `next` (`read::next_active`, `next_active_value`, `next_active_filtered`) ranks a plugin with such keys from a new `ActiveBoard::ranking` graph: the edges around the active tickets and the live neighbours they reach, closed and archived ones included, deleted ones not. A server fills it with `nxs_fold_ddb::records::with_ranking` for what `read::next_ranking_reads(cfg)` names; `read::with_ranking` reads the same graph from a store. Without the graph, `next_active` still refuses such a plugin. Everything is additive: `ActiveBoard` gains a field (it is `#[non_exhaustive]`), and `RankingReads`, `RankingGraph`, `Neighbour`, `next_ranking_reads` and `with_ranking` are new. For a server's operators: the fold's index revision goes from 2 to 3, so every hosted stream is not current and is folded again on its next run. Its records reads (`active_board`, `with_ranking`, `list_board`) answer `NotCurrent` from the moment the run clears it and `Refolding` until the refold has reached the relay position the clear named — never a half-folded board. `Folder::clear` now takes that position (`clear(table, until)`) and leaves a sentinel watermark instead of deleting it. Roll the new build out to every fold run at once: a build with older rules refolds a newer stream back. The `.adj#` adjacency entries gain attributes; the one table-contract change of this release, the `typed` index, is described with the hosted `list`. [de] -Ein gehostetes Brett rankt `next` jetzt nach abgeleiteten Ranking-Schlüsseln (`related`, `count`, `sum`) genau so wie ein lokales Replikat. Das speicherfreie `next` (`read::next_active`, `next_active_value`, `next_active_filtered`) rankt ein Plugin mit solchen Schlüsseln aus einem neuen Graphen `ActiveBoard::ranking`: den Kanten um die aktiven Tickets und den lebenden Nachbarn, die sie erreichen, geschlossene und archivierte eingeschlossen, gelöschte nicht. Ein Server füllt ihn mit `nxs_fold_ddb::records::with_ranking` für das, was `read::next_ranking_reads(cfg)` nennt; `read::with_ranking` liest denselben Graphen aus einem Speicher. Ohne den Graphen lehnt `next_active` ein solches Plugin weiterhin ab. Alles ist additiv: `ActiveBoard` bekommt ein Feld (es ist `#[non_exhaustive]`), und `RankingReads`, `RankingGraph`, `Neighbour`, `next_ranking_reads` und `with_ranking` sind neu. Für den Betrieb eines Servers: Die Index-Revision des Folds steigt von 2 auf 3, daher ist jeder gehostete Stream nicht aktuell und wird bei seinem nächsten Lauf neu gefaltet; bis dahin antwortet sein Records-Lesen mit `NotCurrent`. Die `.adj#`-Adjazenzeinträge bekommen Attribute; die eine Änderung des Tabellenvertrags in diesem Release, der Index `typed`, ist bei der gehosteten `list` beschrieben. +Ein gehostetes Brett rankt `next` jetzt nach abgeleiteten Ranking-Schlüsseln (`related`, `count`, `sum`) genau so wie ein lokales Replikat. Das speicherfreie `next` (`read::next_active`, `next_active_value`, `next_active_filtered`) rankt ein Plugin mit solchen Schlüsseln aus einem neuen Graphen `ActiveBoard::ranking`: den Kanten um die aktiven Tickets und den lebenden Nachbarn, die sie erreichen, geschlossene und archivierte eingeschlossen, gelöschte nicht. Ein Server füllt ihn mit `nxs_fold_ddb::records::with_ranking` für das, was `read::next_ranking_reads(cfg)` nennt; `read::with_ranking` liest denselben Graphen aus einem Speicher. Ohne den Graphen lehnt `next_active` ein solches Plugin weiterhin ab. Alles ist additiv: `ActiveBoard` bekommt ein Feld (es ist `#[non_exhaustive]`), und `RankingReads`, `RankingGraph`, `Neighbour`, `next_ranking_reads` und `with_ranking` sind neu. Für den Betrieb eines Servers: Die Index-Revision des Folds steigt von 2 auf 3, daher ist jeder gehostete Stream nicht aktuell und wird bei seinem nächsten Lauf neu gefaltet. Seine Records-Lesezugriffe (`active_board`, `with_ranking`, `list_board`) antworten ab dem Moment, in dem der Lauf ihn leert, mit `NotCurrent` und mit `Refolding`, bis das Neufalten die Relay-Position erreicht hat, die das Leeren genannt hat — nie mit einem halb gefalteten Brett. `Folder::clear` nimmt diese Position jetzt entgegen (`clear(table, until)`) und hinterlässt ein Sentinel-Watermark, statt es zu löschen. Den neuen Build in alle Fold-Läufe zugleich ausrollen: Ein Build mit älteren Regeln faltet einen neueren Stream zurück. Die `.adj#`-Adjazenzeinträge bekommen Attribute; die eine Änderung des Tabellenvertrags in diesem Release, der Index `typed`, ist bei der gehosteten `list` beschrieben. diff --git a/changes/xkr7-typed-index.md b/changes/xkr7-typed-index.md index 31fb4a1..281a6b5 100644 --- a/changes/xkr7-typed-index.md +++ b/changes/xkr7-typed-index.md @@ -3,6 +3,6 @@ type: added facade: changed --- [en] -A hosted board now answers `list` ranked by the plugin's `[ranking.list]`, derived ranking keys included, exactly as a local replica does. The facade names what to read for a query (`read::list_reads`, a new `ListReads`), a server reads it into a new `ListBoard` with `nxs_fold_ddb::records::list_board`, and `read::list_board_page` / `read::list_board_value` filter (type, container, label, status), rank and project it with the code `list` runs locally; `read::list_board` reads the same board from a store. The hosted list is not paged: `limit` cuts it with the total disclosed, and `paginate` or a token is refused, as for the hosted `next`. A hosted `next` whose keys read the rank `[ranking.list]` gives a type now ranks too instead of being refused: `RankingReads` gains `list_types` and `RankingGraph` gains `listed` (both `#[non_exhaustive]`, so additive). For a server's operators, the table contract changes: a third sparse index, GSI `typed` — partition key `nxf_typed` (string, `#`), sort key `sk` (string), projection ALL — set by the fold on every live item row (closed and archived included, removed on delete). Add the index to the stack first, then deploy the read; a fold against a table without the index writes the attribute harmlessly. It ships in fold index revision 3 together with the self-describing adjacency entries, so one refold covers both. +A hosted board now answers `list` ranked by the plugin's `[ranking.list]`, derived ranking keys included, exactly as a local replica does. The facade names what to read for a query (`read::list_reads`, a new `ListReads`), a server reads it into a new `ListBoard` with `nxs_fold_ddb::records::list_board`, and `read::list_board_page` / `read::list_board_value` filter (type, container, label, status), rank and project it with the code `list` runs locally; `read::list_board` reads the same board from a store. The hosted list is not paged: `limit` cuts it with the total disclosed, and `paginate` or a token is refused, as for the hosted `next`. A list that is not ranked costs its page, not the stream: `ListReads::select` (a new `ListSelect`) lets the server keep and cut on the item rows before it reads any join (`read::list_select`, and `ListBoard::cut` with a new `ListCut` carries the total counted before the cut), and the server reads its point reads by `BatchGetItem`; a ranked list reads its types' whole `list` scope, which its rank is computed over. A hosted `next` whose keys read the rank `[ranking.list]` gives a type now ranks too instead of being refused: `RankingReads` gains `list_types` and `RankingGraph` gains `listed` (both `#[non_exhaustive]`, so additive). For a server's operators, the table contract changes: a third sparse index, GSI `typed` — partition key `nxf_typed` (string, `#`), sort key `sk` (string), projection ALL — set by the fold on every live item row (closed and archived included, removed on delete). Add the index to the stack first, then deploy the read; a fold against a table without the index writes the attribute harmlessly. It ships in fold index revision 3 together with the self-describing adjacency entries, so one refold covers both. For an embedder of `nxs-fold-ddb`: the `Table` trait gains `get_many` (one `BatchGetItem` per 100 keys, unprocessed keys sent again), `RecordsError` gains `Refolding`, `Watermark` gains `refold_until`, `board::reindex` returns the items whose index attributes did not fit, and an item op is refused (`Folded::Refused`) when its row would pass 400 KB together with its index attributes, at one `GetItem` more per op, instead of silently dropping off an index. [de] -Ein gehostetes Brett beantwortet `list` jetzt nach dem `[ranking.list]` des Plugins gerankt, abgeleitete Ranking-Schlüssel eingeschlossen, genau so wie ein lokales Replikat. Die Fassade nennt, was für eine Anfrage zu lesen ist (`read::list_reads`, ein neues `ListReads`), ein Server liest es mit `nxs_fold_ddb::records::list_board` in ein neues `ListBoard`, und `read::list_board_page` / `read::list_board_value` filtern (Typ, Container, Label, Status), ranken und projizieren es mit dem Code, den `list` lokal ausführt; `read::list_board` liest dasselbe Brett aus einem Speicher. Die gehostete Liste wird nicht geblättert: `limit` schneidet sie mit offengelegter Gesamtzahl, `paginate` oder ein Token wird abgelehnt, wie beim gehosteten `next`. Ein gehostetes `next`, dessen Schlüssel den Rang lesen, den `[ranking.list]` einem Typ gibt, rankt jetzt ebenfalls, statt abgelehnt zu werden: `RankingReads` bekommt `list_types` und `RankingGraph` bekommt `listed` (beide `#[non_exhaustive]`, also additiv). Für den Betrieb eines Servers ändert sich der Tabellenvertrag: ein dritter dünn besetzter Index, GSI `typed` — Partitionsschlüssel `nxf_typed` (String, `#`), Sortierschlüssel `sk` (String), Projektion ALL —, den der Fold an jeder lebenden Item-Zeile setzt (geschlossene und archivierte eingeschlossen, beim Löschen entfernt). Zuerst den Index in den Stack aufnehmen, dann das Lesen ausrollen; ein Fold gegen eine Tabelle ohne den Index schreibt das Attribut folgenlos. Er kommt mit der Fold-Index-Revision 3 zusammen mit den selbstbeschreibenden Adjazenzeinträgen, ein Neufalten deckt also beides ab. +Ein gehostetes Brett beantwortet `list` jetzt nach dem `[ranking.list]` des Plugins gerankt, abgeleitete Ranking-Schlüssel eingeschlossen, genau so wie ein lokales Replikat. Die Fassade nennt, was für eine Anfrage zu lesen ist (`read::list_reads`, ein neues `ListReads`), ein Server liest es mit `nxs_fold_ddb::records::list_board` in ein neues `ListBoard`, und `read::list_board_page` / `read::list_board_value` filtern (Typ, Container, Label, Status), ranken und projizieren es mit dem Code, den `list` lokal ausführt; `read::list_board` liest dasselbe Brett aus einem Speicher. Die gehostete Liste wird nicht geblättert: `limit` schneidet sie mit offengelegter Gesamtzahl, `paginate` oder ein Token wird abgelehnt, wie beim gehosteten `next`. Eine nicht gerankte Liste kostet ihre Seite, nicht den Stream: `ListReads::select` (ein neues `ListSelect`) lässt den Server anhand der Item-Zeilen behalten und schneiden, bevor er einen Join liest (`read::list_select`; `ListBoard::cut` mit einem neuen `ListCut` trägt die vor dem Schnitt gezählte Gesamtzahl), und der Server liest seine Punktzugriffe per `BatchGetItem`; eine gerankte Liste liest den ganzen `list`-Bereich ihrer Typen, über den ihr Rang berechnet wird. Ein gehostetes `next`, dessen Schlüssel den Rang lesen, den `[ranking.list]` einem Typ gibt, rankt jetzt ebenfalls, statt abgelehnt zu werden: `RankingReads` bekommt `list_types` und `RankingGraph` bekommt `listed` (beide `#[non_exhaustive]`, also additiv). Für den Betrieb eines Servers ändert sich der Tabellenvertrag: ein dritter dünn besetzter Index, GSI `typed` — Partitionsschlüssel `nxf_typed` (String, `#`), Sortierschlüssel `sk` (String), Projektion ALL —, den der Fold an jeder lebenden Item-Zeile setzt (geschlossene und archivierte eingeschlossen, beim Löschen entfernt). Zuerst den Index in den Stack aufnehmen, dann das Lesen ausrollen; ein Fold gegen eine Tabelle ohne den Index schreibt das Attribut folgenlos. Er kommt mit der Fold-Index-Revision 3 zusammen mit den selbstbeschreibenden Adjazenzeinträgen, ein Neufalten deckt also beides ab. Für Einbettende von `nxs-fold-ddb`: Das Trait `Table` bekommt `get_many` (ein `BatchGetItem` je 100 Schlüssel, unverarbeitete Schlüssel werden erneut gesendet), `RecordsError` bekommt `Refolding`, `Watermark` bekommt `refold_until`, `board::reindex` gibt die Items zurück, deren Index-Attribute nicht passten, und eine Item-Op wird abgelehnt (`Folded::Refused`), wenn ihre Zeile zusammen mit ihren Index-Attributen 400 KB überschritte — zum Preis eines `GetItem` mehr je Op —, statt still aus einem Index zu fallen. diff --git a/crates/facade/src/read.rs b/crates/facade/src/read.rs index 6f3ec09..0c5da3b 100644 --- a/crates/facade/src/read.rs +++ b/crates/facade/src/read.rs @@ -2592,7 +2592,15 @@ fn date_cmp(a: &ItemRow, b: &ItemRow, field: &str, dir: Dir) -> std::cmp::Orderi fn sort_cmp(cfg: &PluginConfig, a: &ItemRow, b: &ItemRow, key: SortKey) -> std::cmp::Ordering { match key { SortKey::Rank => rank_cmp(cfg, a, b), - SortKey::Id => a.id.cmp(&b.id), + _ => row_cmp(a, b, key), + } +} + +/// [`sort_cmp`] for every key but `rank`: the orders that read the row alone, and no plugin. +fn row_cmp(a: &ItemRow, b: &ItemRow, key: SortKey) -> std::cmp::Ordering { + match key { + // Never asked: `sort_cmp` ranks, and a `ListSelect` never orders by rank. + SortKey::Rank | SortKey::Id => a.id.cmp(&b.id), SortKey::Defer => date_cmp(a, b, "defer_until", Dir::Asc), SortKey::Closed => date_cmp(a, b, "closed_at", Dir::Desc), SortKey::Archived => date_cmp(a, b, "archived", Dir::Desc), @@ -3202,6 +3210,109 @@ pub struct ListReads { /// the active tickets when `next` is set, and its own listed items /// ([`RankingReads::list_types`]). pub ranking: Option, + /// What a `list` that is NOT ranked keeps and where it cuts (review of PR #55): its order and + /// its row predicates read the item rows alone, so a reader keeps and cuts BEFORE it reads the + /// joins a record carries, and reads them for the kept page only ([`list_select`]). `None` for + /// `sort = rank`: a rank is computed over the whole `list` scope of a type — every live, + /// unarchived item of it, whatever the query's status, filter or limit + /// (docs/specs/derived-ranking-keys.md §12, decision 2) — so a ranked list reads the scope + /// whole. + pub select: Option, +} + +/// The query of a `list` that is not ranked, as a reader applies it before the joins +/// ([`ListReads::select`], [`list_select`]). +/// +/// `#[non_exhaustive]`: only [`list_reads`] builds one. +#[derive(Debug, Clone, PartialEq, Eq)] +#[non_exhaustive] +pub struct ListSelect { + /// Only items with this core status. + pub status: Option, + /// The query's filter: its types read the row, its label the item's labels, its container the + /// item's current parent and `contributes_to` targets. + pub filter: NextFilter, + /// The order — any key but `rank` ([`ListSelect::sort`]); it reads the row alone. + pub(crate) sort: SortKey, + /// The cut; `None`: every kept item. + pub limit: Option, +} + +impl ListSelect { + /// The order: any key but `rank`, which reads the row alone. + pub fn sort(&self) -> SortKey { + self.sort + } + + /// Whether an item passes what its row alone decides: live, unarchived, of the status and of a + /// type the filter names. A reader asks it first, and reads no join of an item it refuses. + pub fn keeps_row(&self, item: &ItemRow) -> bool { + item.deleted.as_deref() != Some("1") + && item.archived.is_none() + && self + .status + .as_deref() + .is_none_or(|s| item.status.as_deref() == Some(s)) + && (self.filter.types.is_empty() + || item + .item_type + .as_deref() + .is_some_and(|t| self.filter.types.iter().any(|f| f == t))) + } + + /// Whether the filter reads an item's labels: a reader reads them before [`list_select`]. + pub fn reads_labels(&self) -> bool { + self.filter.label.is_some() + } + + /// Whether the filter reads an item's current parent (`belongs_to`) and `contributes_to` + /// targets: a reader reads them before [`list_select`]. + pub fn reads_edges(&self) -> bool { + self.filter.within.is_some() + } +} + +/// Where a reader cut a [`ListBoard`] ([`list_select`]): the query it kept and cut for, and how many +/// items it kept before the cut — the total the page discloses. +/// +/// `#[non_exhaustive]`: only [`list_select`] builds one. +#[derive(Debug, Clone, PartialEq, Eq)] +#[non_exhaustive] +pub struct ListCut { + pub select: ListSelect, + pub total: usize, +} + +/// The items of `items` that `select` keeps — [`ListSelect::keeps_row`] and the filter's one +/// membership rule ([`NextFilter`]) — in `select.sort` order, cut to `select.limit`, with the +/// [`ListCut`] to put on the board ([`ListBoard::cut`]). What [`list_board_page`] keeps, orders and +/// cuts, so a reader that reads the joins of the kept items only answers the same page. +/// +/// Every item must carry its labels when [`ListSelect::reads_labels`], and its `belongs_to` and +/// `contributes_to` when [`ListSelect::reads_edges`]; no other join is read here. +pub fn list_select(select: &ListSelect, items: Vec) -> (Vec, ListCut) { + let mut kept: Vec = items + .into_iter() + .filter(|i| { + select.keeps_row(&i.item) && select.filter.keeps(&i.item, &i.labels, &i.contributes_to) + }) + .collect(); + // Ties by id: `list_board_page` sorts the board's items stably from id order, so ties are cut + // alike. + kept.sort_by(|a, b| { + row_cmp(&a.item, &b.item, select.sort).then_with(|| a.item.id.cmp(&b.item.id)) + }); + let total = kept.len(); + if let Some(n) = select.limit { + kept.truncate(n); + } + ( + kept, + ListCut { + select: select.clone(), + total, + }, + ) } /// One listed item and the joins its `list` record and the filters read (6j6v.xkr7) — the @@ -3266,6 +3377,10 @@ pub struct ListBoard { pub active: Option, /// The graph the derived keys read ([`ListReads::ranking`]). pub ranking: Option, + /// Set when the reader kept and cut `items` for a [`ListReads::select`] ([`list_select`]): + /// `items` then holds the kept page only, and the cut the total before it. `None`: `items` + /// holds every listed item. + pub cut: Option, } impl ListBoard { @@ -3279,6 +3394,7 @@ impl ListBoard { items: items.into_iter().map(|i| (i.item.id.clone(), i)).collect(), active: None, ranking: None, + cut: None, } } } @@ -3312,7 +3428,6 @@ pub fn list_board(store: &Store, reads: &ListReads) -> Result { let mut listed = Vec::with_capacity(items.len()); for item in items { let id = item.id.clone(); - let parent_closed = parent_closed_reason(store, &item)?; let (created_at, updated_at) = ts.remove(&id).unwrap_or_default(); let mut i = ListItem::new(item); i.labels = labels.remove(&id).unwrap_or_default(); @@ -3320,10 +3435,22 @@ pub fn list_board(store: &Store, reads: &ListReads) -> Result { i.created_at = created_at; i.updated_at = updated_at; i.contributes_to = contributes.remove(&id).unwrap_or_default(); - i.parent_closed = parent_closed; listed.push(i); } + // Kept and cut as a server keeps and cuts it (`list_select`), so the two boards compare whole. + let cut = match &reads.select { + Some(select) => { + let (kept, cut) = list_select(select, listed); + listed = kept; + Some(cut) + } + None => None, + }; + for i in &mut listed { + i.parent_closed = parent_closed_reason(store, &i.item)?; + } let mut board = ListBoard::new(reads.types.clone(), listed); + board.cut = cut; let graph_reads = reads.ranking.clone().unwrap_or_default(); if reads.next { let selection = nexus_flow_core::graph::select(store.connection())?; @@ -3367,7 +3494,9 @@ pub fn list_board(store: &Store, reads: &ListReads) -> Result { /// Not paged: the store path's page cache is machine-local, and a server's paging is an open /// decision (6j6v.15ed) — as for [`next_active_filtered`], a server cuts this list itself. So /// `validation` for `q.paginate` or a token; and for a malformed `q.now`, a type filter (or none) -/// the board does not list every item for, and a board read for less than the ranking needs. +/// the board does not list every item for, a board read for less than the ranking needs, and a +/// board kept and cut ([`ListBoard::cut`]) for another query than `q`. A cut board answers the +/// total its reader counted before the cut. pub fn list_board_page(cfg: &PluginConfig, board: &ListBoard, q: &ListQuery) -> Result { if q.paginate || q.token.is_some() { return Err(NxfError::validation( @@ -3390,6 +3519,14 @@ pub fn list_board_page(cfg: &PluginConfig, board: &ListBoard, q: &ListQuery) -> `list_reads` for this query", )); } + if let Some(cut) = &board.cut { + if list_reads(cfg, q).select.as_ref() != Some(&cut.select) { + return Err(NxfError::validation( + "the list board was kept and cut for another query — read it with `list_reads` \ + for this query", + )); + } + } let mut items: Vec = board .items .values() @@ -3411,7 +3548,12 @@ pub fn list_board_page(cfg: &PluginConfig, board: &ListBoard, q: &ListQuery) -> } else { order_by(cfg, &mut items, key); } - Ok(truncate_next(items, q.limit)) + let mut page = truncate_next(items, q.limit); + if let Some(cut) = &board.cut { + // The reader kept the page only: the total is the one it counted before its cut. + page.total = cut.total; + } + Ok(page) } /// [`list_board_page`] as the `list` JSON envelope `{"items", "total"}` — the value diff --git a/crates/facade/src/read/derived.rs b/crates/facade/src/read/derived.rs index 7c8d8ad..0575049 100644 --- a/crates/facade/src/read/derived.rs +++ b/crates/facade/src/read/derived.rs @@ -35,8 +35,8 @@ //! may read one too ([`compute_active`]). What a `list` reads is [`reads_for_list`]. use super::{ - lanes, mentioners_of, resolve_items, ActiveBoard, ListBoard, ListQuery, ListReads, - RankingGraph, RankingReads, SortKey, + lanes, mentioners_of, resolve_items, ActiveBoard, ListBoard, ListQuery, ListReads, ListSelect, + RankingGraph, RankingReads, SortKey, DEFAULT_SORT_LIST, }; use crate::error::{NxfError, Result}; use crate::plugin::{Dir, Nulls, Pick, PluginConfig, RankKey, RankOf, RankSpec, Verb}; @@ -382,9 +382,9 @@ pub(super) fn compute( /// the graph's listed items (6j6v.xkr7). `None` when the policy has no derived key. /// /// The same [`Context`] evaluates the keys as on the store path; only where its edges, rows, -/// labels and custom values come from differs ([`Context::read_graph`]). `validation` when the graph -/// was read for less than the plan needs — a `[ranking.list]` type of the chain missing from -/// [`RankingReads::list_types`] among it. +/// labels and custom values come from differs ([`Context::read_graph`]). The caller, +/// [`next_active`](super::next_active), has refused a graph read for less than +/// [`next_ranking_reads`](super::next_ranking_reads) names, which covers every plan of the section. pub(super) fn compute_active( cfg: &PluginConfig, board: &ActiveBoard, @@ -394,13 +394,9 @@ pub(super) fn compute_active( let Some(plan) = plan(cfg, Verb::Next, candidates) else { return Ok(None); }; - let needed = plan.reads(|_| false); - let graph = graph_for( - Some(graph), - &needed, - graph, - &format!("plugin '{}' (`next`)", cfg.name), - )?; + // `next_active` admitted the graph only when it covers `next_ranking_reads`, which gathers the + // reads of every order of the section and of its chain — so of this plan's too. + debug_assert!(graph.reads.covers(&plan.reads(|_| false))); let rows = |id: &str| active_joined(board, graph, id); let listed = listed_rows(graph, rows); let node_rows = plan.node_rows(candidates, &listed); @@ -504,9 +500,22 @@ pub(super) fn reads_for_list(cfg: &PluginConfig, q: &ListQuery) -> ListReads { types: types.clone(), next: false, ranking: None, + select: None, }; + // Any order but `rank` reads the item rows alone: the reader may keep and cut before it reads + // an item's joins (review of PR #55, Integrity #1). + let sort = q.sort.unwrap_or(DEFAULT_SORT_LIST); + if sort != SortKey::Rank { + reads.select = Some(ListSelect { + status: q.status.clone(), + filter: q.filter.clone(), + sort, + limit: q.limit, + }); + return reads; + } let spec = cfg.ranking.spec(Verb::List); - if q.sort != Some(SortKey::Rank) || !spec.has_derived() { + if !spec.has_derived() { return reads; } let orders: Vec<&[RankKey]> = match &types { diff --git a/crates/facade/tests/derived_ranking.rs b/crates/facade/tests/derived_ranking.rs index c49f900..d5ad251 100644 --- a/crates/facade/tests/derived_ranking.rs +++ b/crates/facade/tests/derived_ranking.rs @@ -730,6 +730,20 @@ fn the_store_free_list_refuses_what_its_board_was_not_read_for() { short.next = false; let msg = refused(&persons, &read::list_board(&s, &short).unwrap()); assert!(msg.contains("active board"), "{msg}"); + // A list that is not ranked is kept and cut before its joins: its board holds the page of ONE + // query, and answers no other — not another status, not another limit (review of PR #55). + let closed = ListQuery::new().status("closed").limit(1); + let cut = read::list_board(&s, &read::list_reads(&cfg, &closed)).unwrap(); + assert!(cut.cut.is_some() && cut.items.len() <= 1); + read::list_board_page(&cfg, &cut, &closed).unwrap(); + for other in [ + ListQuery::new().status("open").limit(1), + ListQuery::new().status("closed").limit(2), + ListQuery::new().status("closed"), + ] { + let msg = refused(&other, &cut); + assert!(msg.contains("kept and cut for another query"), "{msg}"); + } } // ---- each kind over hand-built boards (§9 "per kind") ------------------------------------- diff --git a/crates/fold-ddb/Cargo.toml b/crates/fold-ddb/Cargo.toml index 27dc171..d1d1e7b 100644 --- a/crates/fold-ddb/Cargo.toml +++ b/crates/fold-ddb/Cargo.toml @@ -2,7 +2,7 @@ name = "nxs-fold-ddb" version.workspace = true edition = "2021" -description = "Folding on a server (6j6v.vvw6): the reducers' described changes carried out against DynamoDB, the board's active and dated indexes, next/blocked over them, snapshots as tables plus watermark" +description = "Folding on a server (6j6v.vvw6): the reducers' described changes carried out against DynamoDB, the board's active, dated and typed indexes, next/blocked and a ranked list over them, snapshots as tables plus watermark" publish = false [features] diff --git a/crates/fold-ddb/src/board.rs b/crates/fold-ddb/src/board.rs index d43f5d0..8fbc37c 100644 --- a/crates/fold-ddb/src/board.rs +++ b/crates/fold-ddb/src/board.rs @@ -86,8 +86,8 @@ use crate::layout::{ link_adjacency_key, row_key, table_prefix, typed_partition, ACTIVE, DATED, DATED_AT, MAX_KEY_BYTES, SK, TYPED, }; -use crate::table::{text, Dated, Row, Table}; -use crate::write::{Cond, Write}; +use crate::table::{text, Dated, Outcome, Row, Table}; +use crate::write::{attribute_bytes, Cond, Write}; use nexus_flow_core::graph::{Board, Edge, Ticket}; use nexus_flow_core::model::EdgeKind; use nxs_foundation::change::{Cell, Change, Effect}; @@ -294,8 +294,47 @@ pub fn dated_key(at: &str, id: &str) -> String { } } -/// Keep the indexes after one op's changes were carried out. -pub(crate) async fn after(table: &T, changes: &[Change]) -> Result<(), T::Error> { +/// The index attributes a stored `items` row of ticket `id` carries by the rules above — what +/// [`refresh_item`] leaves on it: `active`, `typed` and the `dated` pair, each only where it +/// applies. +pub(crate) fn index_attributes(stream: &str, id: &str, row: &Row) -> Vec<(String, Cell)> { + let mut cells = Vec::new(); + if is_active(row) { + cells.push((ACTIVE.to_string(), Cell::Text(stream.to_string()))); + } + if let Some(ty) = typed(row) { + cells.push((TYPED.to_string(), Cell::Text(typed_partition(stream, ty)))); + } + if let Some((l, at)) = lane(row) { + cells.push((DATED.to_string(), Cell::Text(l.partition(stream)))); + cells.push((DATED_AT.to_string(), Cell::Text(dated_key(&at, id)))); + } + cells +} + +/// What an `items` row weighs toward `MAX_ITEM_BYTES` once it carries the index attributes its +/// cells call for ([`index_attributes`]) instead of the ones it has — what +/// [`Folder::fold`](crate::fold::Folder::fold) checks before an op writes anything, so a row with +/// no room left for them is refused rather than left off an index (review of PR #55, Integrity +/// #4). +pub(crate) fn indexed_bytes(stream: &str, id: &str, row: &Row) -> usize { + let own: usize = row + .iter() + .filter(|(n, _)| ![ACTIVE, TYPED, DATED, DATED_AT].contains(&n.as_str())) + .map(|(n, c)| attribute_bytes(n, c)) + .sum(); + own + index_attributes(stream, id, row) + .iter() + .map(|(n, c)| attribute_bytes(n, c)) + .sum::() +} + +/// Keep the indexes after one op's changes were carried out. Returns the items whose row could not +/// take its index attributes ([`Outcome::TooLarge`]) — none, when the fold's size check ran first. +pub(crate) async fn after( + table: &T, + changes: &[Change], +) -> Result, T::Error> { let mut items = BTreeSet::new(); let mut added = BTreeSet::new(); let mut removed = BTreeSet::new(); @@ -320,8 +359,11 @@ pub(crate) async fn after(table: &T, changes: &[Change]) -> Result<(), _ => {} } } + let mut unindexed = Vec::new(); for id in items { - refresh_item(table, id).await?; + if refresh_item(table, id).await? == Some(Outcome::TooLarge) { + unindexed.push(id.to_string()); + } } for tag in added { link_edge(table, tag).await?; @@ -332,7 +374,7 @@ pub(crate) async fn after(table: &T, changes: &[Change]) -> Result<(), for (at, tag) in owned_tags { refresh_owned(table, &OWNED[at], tag).await?; } - Ok(()) + Ok(unindexed) } /// Set ticket `id`'s index attributes from its row; when its activity flipped, refresh every edge @@ -341,10 +383,17 @@ pub(crate) async fn after(table: &T, changes: &[Change]) -> Result<(), /// The edges go FIRST, told the ticket's new activity, and the ticket's own flag after: the flag is /// what says whether the activity flipped, so a run that dies in between still sees the flip when /// it folds the op again, and redoes the fan-out (review of PR #24, Integrity #2). -pub(crate) async fn refresh_item(table: &T, id: &str) -> Result<(), T::Error> { +/// +/// Returns the outcome of the write of its index attributes, `None` when nothing changed — a +/// [`Outcome::TooLarge`] means the row is left with the attributes it had, which the caller must +/// report rather than drop (review of PR #55, Integrity #4). +pub(crate) async fn refresh_item( + table: &T, + id: &str, +) -> Result, T::Error> { let sk = item_key(id); let Some(row) = table.get(&sk).await? else { - return Ok(()); + return Ok(None); }; let active = is_active(&row); let was_active = row.contains_key(ACTIVE); @@ -392,9 +441,9 @@ pub(crate) async fn refresh_item(table: &T, id: &str) -> Result<(), T: } } if set.is_empty() && remove.is_empty() { - return Ok(()); + return Ok(None); } - table + let outcome = table .write(&Write { sk, set, @@ -402,7 +451,7 @@ pub(crate) async fn refresh_item(table: &T, id: &str) -> Result<(), T: condition: Some(Cond::RowPresent), }) .await?; - Ok(()) + Ok(Some(outcome)) } /// The cells of edge add `edge` an adjacency entry copies beside its tag and [`REMOVED`] (index @@ -510,11 +559,15 @@ async fn flag_edge( } /// Recompute every index attribute and adjacency entry from the rows — after a snapshot was -/// loaded, or to repair a stream two folders wrote at once. -pub async fn reindex(table: &T) -> Result<(), T::Error> { +/// loaded, or to repair a stream two folders wrote at once. Returns the items whose row could not +/// take its index attributes (past `MAX_ITEM_BYTES` with them), which keep the ones they had. +pub async fn reindex(table: &T) -> Result, T::Error> { + let mut unindexed = Vec::new(); for row in table.query_prefix(&table_prefix("items")).await? { if let Some(id) = text(&row, "id") { - refresh_item(table, id).await?; + if refresh_item(table, id).await? == Some(Outcome::TooLarge) { + unindexed.push(id.to_string()); + } } } for row in table.query_prefix(&table_prefix("edge_adds")).await? { @@ -529,7 +582,7 @@ pub async fn reindex(table: &T) -> Result<(), T::Error> { } } } - Ok(()) + Ok(unindexed) } /// The active tickets — their whole rows — and the board the library derives the lanes from. diff --git a/crates/fold-ddb/src/ddb.rs b/crates/fold-ddb/src/ddb.rs index 323d715..ffd46f1 100644 --- a/crates/fold-ddb/src/ddb.rs +++ b/crates/fold-ddb/src/ddb.rs @@ -1,7 +1,8 @@ //! [`DynamoDbTable`]: one stream's partition of the views table in DynamoDB, and [`create_table`], //! the table the contract in the crate doc describes. //! -//! Every request names the stream's partition key; the board's three index reads name an index. There is +//! Every request names the stream's partition key — a `BatchGetItem` names it in every key; the +//! board's three index reads name an index. There is //! no Scan anywhere in this file, and the `dynamodb-local` CI job records every request a fold //! makes to prove it stays that way. //! @@ -11,17 +12,23 @@ use crate::layout::{ typed_partition, ACTIVE, ACTIVE_INDEX, DATED, DATED_AT, DATED_INDEX, PK, SK, TYPED, TYPED_INDEX, }; -use crate::table::{Dated, Outcome, Row, Table}; +use crate::table::{Dated, Outcome, Row, Table, BATCH_GET_KEYS}; use crate::write::Write; +use aws_sdk_dynamodb::config::AsyncSleep; use aws_sdk_dynamodb::error::{DisplayErrorContext, ProvideErrorMetadata}; use aws_sdk_dynamodb::operation::create_table::builders::CreateTableFluentBuilder; use aws_sdk_dynamodb::types::{ AttributeDefinition, AttributeValue, BillingMode, GlobalSecondaryIndex, KeySchemaElement, - KeyType, Projection, ProjectionType, ScalarAttributeType, + KeyType, KeysAndAttributes, Projection, ProjectionType, ScalarAttributeType, }; use aws_sdk_dynamodb::Client; use nxs_foundation::change::Cell; -use std::collections::HashMap; +use std::collections::{BTreeMap, BTreeSet, HashMap}; + +/// How often [`Table::get_many`] sends the keys a `BatchGetItem` left unprocessed again before it +/// gives up, waiting twice as long each time from [`UNPROCESSED_BACKOFF_MS`]. +const UNPROCESSED_RETRIES: u32 = 8; +const UNPROCESSED_BACKOFF_MS: u64 = 25; /// A request DynamoDB refused, or an item this crate cannot read. #[derive(Debug)] @@ -210,6 +217,66 @@ impl Table for DynamoDbTable { out.item().map(from_item).transpose() } + async fn get_many(&self, sks: &[String]) -> Result, DdbError> { + let keys: Vec<&String> = sks.iter().collect::>().into_iter().collect(); + let mut rows = BTreeMap::new(); + for chunk in keys.chunks(BATCH_GET_KEYS) { + let mut pending: Vec> = + chunk.iter().map(|sk| self.key(sk)).collect(); + let mut retries = 0; + while !pending.is_empty() { + let request = KeysAndAttributes::builder() + .set_keys(Some(pending)) + .consistent_read(true) + .build() + .map_err(err)?; + let out = self + .client + .batch_get_item() + .request_items(&self.table, request) + .send() + .await + .map_err(err)?; + for item in out + .responses() + .and_then(|r| r.get(&self.table)) + .into_iter() + .flatten() + { + let row = from_item(item)?; + if let Some(Cell::Text(sk)) = row.get(SK) { + rows.insert(sk.clone(), row.clone()); + } + } + pending = out + .unprocessed_keys() + .and_then(|u| u.get(&self.table)) + .map(|k| k.keys().to_vec()) + .unwrap_or_default(); + if !pending.is_empty() { + // Throttled, or past the 16 MB a response carries: the rest again, after a + // pause that doubles, as DynamoDB asks of a caller. + if retries == UNPROCESSED_RETRIES { + return Err(DdbError(format!( + "BatchGetItem left {} keys unprocessed after {UNPROCESSED_RETRIES} \ + retries", + pending.len() + ))); + } + if let Some(sleep) = self.client.config().sleep_impl() { + sleep + .sleep(std::time::Duration::from_millis( + UNPROCESSED_BACKOFF_MS << retries, + )) + .await; + } + retries += 1; + } + } + } + Ok(rows) + } + async fn write(&self, write: &Write) -> Result { let e = write.expression(); let mut req = self diff --git a/crates/fold-ddb/src/fold.rs b/crates/fold-ddb/src/fold.rs index 9770de4..5d8d5ad 100644 --- a/crates/fold-ddb/src/fold.rs +++ b/crates/fold-ddb/src/fold.rs @@ -13,7 +13,15 @@ //! //! When the recorded revisions differ from the reducers' — a release changed what the same ops fold //! to — [`Folder::is_current`] says so, and the run starts over: [`Folder::clear`] the stream's -//! rows, then fold from relay position 0. +//! rows, then fold from relay position 0. The clear leaves a sentinel watermark naming the position +//! the refold has to reach, and readers answer `NotCurrent` / `Refolding` until it has. +//! +//! **No mixed fold versions.** `is_current` is equality, not order: a build with OLDER rules that +//! meets a stream folded by newer ones also finds it not current, and clears and refolds it back. +//! Two builds with different revisions folding the same streams — a rolling deploy, a rollback with +//! a newer build still running — would refold every stream back and forth. Roll a revision bump +//! out to every fold run at once, and roll back the same way (a fold run that builds anew beside +//! the served build, as one that keeps a namespace per build, ping-pongs builds instead). //! //! The rules this crate adds beside the reducers' rows — the board's index flags and the entries it //! keeps under a ticket — have a revision of their own, [`INDEX_REVISION`], recorded in the same @@ -37,9 +45,10 @@ use crate::board; use crate::layout::{ - Layout, LayoutError, MAX_ITEM_BYTES, MAX_KEY_BYTES, MAX_PARTITION_BYTES, WATERMARK_KEY, + table_prefix, Layout, LayoutError, MAX_ITEM_BYTES, MAX_KEY_BYTES, MAX_PARTITION_BYTES, PK, SK, + WATERMARK_KEY, }; -use crate::table::{Outcome, Table}; +use crate::table::{text, Outcome, Table}; use crate::write::{plan, Cond, Write}; use nxs_foundation::change::{Cell, Change, Wins}; use nxs_foundation::model::Op; @@ -50,6 +59,9 @@ use std::collections::BTreeMap; /// (see the module doc). No reducer has this domain. pub const INDEX_DOMAIN: &str = "index"; +/// The watermark attribute [`Folder::clear`] sets: [`Watermark::refold_until`]. +pub const REFOLD_UNTIL: &str = "refold_until"; + /// The revision of those rules. 1: the entries under a ticket for its label and thread link adds /// (6j6v.t1ym). 2: the entries under a ticket for its outgoing `contributes_to` edges (6j6v.15ed). /// 3: the adjacency entries copy their edge's ends, kind and coordinate and say whether it is @@ -80,6 +92,10 @@ pub struct Watermark { pub folded_through: i64, /// Each reducer's fold revision, by domain. pub revisions: BTreeMap, + /// Set by [`Folder::clear`]: the relay position a refold has to reach before the stream is + /// served again — readers answer `RecordsError::Refolding` while `folded_through` is below it + /// (review of PR #55, Integrity #2). `None` on a stream never cleared. + pub refold_until: Option, } /// What [`Folder::fold`] did with an op. @@ -167,11 +183,14 @@ impl Folder { /// Fold one op: carry out its changes, then keep the board's indexes. /// /// An op the table cannot hold — a key past `MAX_KEY_BYTES`, a type whose `typed` index - /// partition is past `MAX_PARTITION_BYTES`, a row past `MAX_ITEM_BYTES` — is + /// partition is past `MAX_PARTITION_BYTES`, a row past `MAX_ITEM_BYTES`, an item row that would + /// pass it once it carries its index attributes (`active`, `typed`, `dated`; one `GetItem` per + /// item row the op writes, to weigh it) — is /// [`Folded::Refused`] before it writes anything, rather than failing the batch for ever: the - /// relay keeps it, as it keeps any op a server does not fold. A row that only outgrows the limit - /// through earlier ops refuses the one write that would break it ([`Outcome::TooLarge`]); the - /// op's other writes stand. That last case is the one place the result depends on the order ops + /// relay keeps it, as it keeps any op a server does not fold. An item row that only outgrows + /// the limit through earlier ops is caught by that weighing too, so the whole op is refused; + /// any other row refuses the one write that would break it ([`Outcome::TooLarge`]), and the + /// op's other writes stand. Those cases are the one place the result depends on the order ops /// arrive in — which op's cell made the row too large — and a local replica, which has no item /// limit, holds all of it. pub async fn fold(&self, table: &T, op: &Op) -> Result> { @@ -218,15 +237,34 @@ impl Folder { w.bytes() ))); } + // An item row takes its index attributes in a write of their own, after the row + // (`board::refresh_item`): one the row has no room for would leave the item off an index + // in silence. So the item rows this op writes are weighed first — stored row, this op's + // writes, and the index attributes it would then carry — and the op is refused before it + // writes anything (review of PR #55, Integrity #4). + for (id, bytes) in self.item_bytes(table, &writes).await? { + if bytes > MAX_ITEM_BYTES { + return Ok(refuse(format!( + "item {id} would weigh {bytes} bytes with its index attributes, past the \ + {MAX_ITEM_BYTES} an item may have" + ))); + } + } let mut too_large = Vec::new(); for write in &writes { if table.write(write).await.map_err(FoldError::Table)? == Outcome::TooLarge { too_large.push(write.sk.clone()); } } - board::after(table, &changes) - .await - .map_err(FoldError::Table)?; + // Past the check above only where DynamoDB counts a few bytes more than `attribute_bytes`: + // reported, never dropped. + too_large.extend( + board::after(table, &changes) + .await + .map_err(FoldError::Table)? + .into_iter() + .map(|id| format!("the index attributes of item {id}")), + ); if too_large.is_empty() { Ok(Folded::Folded) } else { @@ -237,6 +275,50 @@ impl Folder { } } + /// What each item row `writes` touch would weigh after them, with the index attributes it + /// would then carry ([`board::indexed_bytes`]): one `GetItem` per such row. Every write is + /// counted as applied, its condition aside — so a write that would be kept can only make the + /// estimate larger, never let a row past the limit. + async fn item_bytes( + &self, + table: &T, + writes: &[Write], + ) -> Result, FoldError> { + let prefix = table_prefix("items"); + let mut rows: BTreeMap<&str, crate::table::Row> = BTreeMap::new(); + for write in writes.iter().filter(|w| w.sk.starts_with(&prefix)) { + if !rows.contains_key(write.sk.as_str()) { + let stored = table + .get(&write.sk) + .await + .map_err(FoldError::Table)? + .unwrap_or_else(|| { + crate::table::Row::from([ + (PK.to_string(), Cell::Text(table.stream().to_string())), + (SK.to_string(), Cell::Text(write.sk.clone())), + ]) + }); + rows.insert(write.sk.as_str(), stored); + } + if let Some(row) = rows.get_mut(write.sk.as_str()) { + for (a, c) in &write.set { + row.insert(a.clone(), c.clone()); + } + for a in &write.remove { + row.remove(a); + } + } + } + Ok(rows + .values() + .map(|row| { + let id = text(row, "id").unwrap_or_default().to_string(); + let bytes = board::indexed_bytes(table.stream(), &id, row); + (id, bytes) + }) + .collect()) + } + /// Fold a batch of ops pulled from the relay — each with its relay position — and move the /// watermark to the highest position among them. Returns the ops it refused, for the caller to /// log; they do not hold the watermark back. @@ -287,6 +369,36 @@ impl Folder { .map_err(FoldError::Table) } + /// A watermark that names no revision — so no reader serves the stream ([`Folder::clear`], + /// `snapshot::import`) — at `folded_through`, with the position a refold must reach. + pub(crate) fn sentinel(folded_through: i64, until: Option) -> Write { + let mut set = vec![("folded_through".to_string(), Cell::Int(folded_through))]; + set.extend(until.map(|u| (REFOLD_UNTIL.to_string(), Cell::Int(u)))); + Write { + sk: WATERMARK_KEY.to_string(), + set, + remove: Vec::new(), + condition: None, + } + } + + /// This folder's watermark at `through`, written whatever the stored one says, and without a + /// refold position — what `snapshot::import` ends with. + pub(crate) fn stamp(&self, through: i64) -> Write { + let mut set = vec![("folded_through".to_string(), Cell::Int(through))]; + set.extend( + self.revisions() + .into_iter() + .map(|(domain, rev)| (format!("revision_{domain}"), Cell::Int(rev))), + ); + Write { + sk: WATERMARK_KEY.to_string(), + set, + remove: vec![REFOLD_UNTIL.to_string()], + condition: None, + } + } + /// The stream's watermark, `None` before its first batch. pub async fn watermark(table: &T) -> Result, T::Error> { Ok(table.get(WATERMARK_KEY).await?.map(|row| { @@ -294,6 +406,7 @@ impl Folder { for (attr, cell) in row { match (attr.as_str(), cell) { ("folded_through", Cell::Int(v)) => mark.folded_through = v, + (REFOLD_UNTIL, Cell::Int(v)) => mark.refold_until = Some(v), (a, Cell::Int(v)) if a.starts_with("revision_") => { mark.revisions.insert(a["revision_".len()..].to_string(), v); } @@ -310,18 +423,42 @@ impl Folder { mark.revisions == self.revisions() } - /// Delete every row of the stream — the views, the adjacency entries and the watermark, the - /// watermark LAST: a clear that dies half way leaves a stream whose watermark still names the old - /// revisions, so the next run clears it again instead of taking the leftovers for a fresh stream - /// (review of PR #24, Integrity #3). - pub async fn clear(table: &T) -> Result<(), T::Error> { - for row in table.query_prefix("").await? { - if let Some(Cell::Text(sk)) = row.get(crate::layout::SK) { + /// Delete every row of the stream — the views and the adjacency entries — for a refold from + /// position 0 that has to reach relay position `until` (the stream's head when the refold + /// begins) before the stream is served again. + /// + /// The watermark is not deleted: it is rewritten FIRST, as a sentinel that names no revision, + /// `folded_through` 0 and `refold_until` = `until`. So readers answer `NotCurrent` from the + /// moment the clear begins (review of PR #55, Integrity #2) — a cleared, half-cleared or + /// half-refolded stream is never served as if whole — and a clear that dies half way leaves a + /// watermark that is not current, so the next run clears it again instead of taking the + /// leftovers for a fresh stream (review of PR #24, Integrity #3). The refold's first batch + /// writes this build's revisions ([`Folder::advance`]), and from then on the readers answer + /// `Refolding` until `folded_through` reaches `until`; nothing has to be written for that, so a + /// refold that dies and is resumed, or folds a batch twice, ends served all the same. `until` + /// 0 serves the stream from its first refolded batch on, as before revisions were gated. + pub async fn clear(table: &T, until: i64) -> Result<(), T::Error> { + let rows = table.query_prefix("").await?; + let stale: Vec = rows + .iter() + .find(|r| text(r, crate::layout::SK) == Some(WATERMARK_KEY)) + .map(|mark| { + mark.keys() + .filter(|a| a.starts_with("revision_")) + .cloned() + .collect() + }) + .unwrap_or_default(); + let mut sentinel = Folder::sentinel(0, Some(until)); + sentinel.remove = stale; + table.write(&sentinel).await?; + for row in rows { + if let Some(sk) = text(&row, crate::layout::SK) { if sk != WATERMARK_KEY { table.delete(sk).await?; } } } - table.delete(WATERMARK_KEY).await + Ok(()) } } diff --git a/crates/fold-ddb/src/lib.rs b/crates/fold-ddb/src/lib.rs index 05e279d..59e2c23 100644 --- a/crates/fold-ddb/src/lib.rs +++ b/crates/fold-ddb/src/lib.rs @@ -76,7 +76,9 @@ //! # Cost //! //! A change is one `UpdateItem`. An item field op makes three changes (the cell and the ticket's -//! two instants) plus one `GetItem` for the index, and one more write when the ticket's lanes move. +//! two instants) plus two `GetItem`s — one before, to weigh the row with the index attributes it +//! would then carry against the 400 KB limit, one after, for the index — and one more write when +//! the ticket's lanes move. //! A flip of a ticket's activity adds one Query and, per edge touching it, at most three reads and //! one write. A label or thread link add, or its remove, adds two `GetItem`s and at most one write //! for the entry under the ticket (`board`, "Who owns a label or a thread link"). Reading the @@ -84,9 +86,13 @@ //! the selection, at most [`records::READ_CONCURRENCY`] tickets at a time ([`records`]). Folding an //! edge remove reads the add and the remove and rewrites the entry under each end. The graph the //! derived ranking keys read ([`records::with_ranking`]) costs one Query per active ticket and, per -//! inactive neighbour, one `GetItem` plus up to three Queries. A hosted `list` -//! ([`records::list_board`]) costs one `typed` Query per listed type and `4n + 1 + p` requests for -//! `n` listed items, as the records read. The `typed` index costs a fold at most one write per item +//! inactive neighbour, up to three Queries, their rows read by `BatchGetItem` (100 keys a request). +//! A hosted `list` ([`records::list_board`]) costs one `typed` Query per listed type; one that is +//! not ranked keeps and cuts its rows first and reads three Queries per row of its page plus one +//! `BatchGetItem` for their timestamps, however many items the stream holds; a ranked one reads +//! three Queries per live item of its types — the scope its rank is computed over — and its point +//! reads by `BatchGetItem`. No read caps its own requests: a server bounds one request by the +//! table it hands in (E4 spec §5.1). The `typed` index costs a fold at most one write per item //! op, and only when the op gains, changes or loses the item's partition — a create, a retype, a //! delete — in the same write as the item's other index attributes when they move too. No request //! is a Scan: every read names its partition key, the board's through an index. The @@ -97,7 +103,10 @@ //! What this crate writes beside the reducers' rows — index flags, adjacency entries, the entries //! under a ticket — has its own revision in the watermark ([`fold::INDEX_REVISION`]). A stream //! folded by an older one is not current ([`fold::Folder::is_current`]) and is refolded the way a -//! stream of an older reducer revision is; [`records::active_board`] refuses it until then. +//! stream of an older reducer revision is. Every reader in [`records`] refuses it from the moment +//! the clear begins until the refold reaches the head the clear named ([`fold::Folder::clear`]): +//! `NotCurrent` while the watermark names no revision, `Refolding` after. Fold versions are not +//! mixed: a build with older rules meets a newer stream as not current, too (see [`fold`]). pub mod board; pub mod chunks; diff --git a/crates/fold-ddb/src/mem.rs b/crates/fold-ddb/src/mem.rs index 155b9b1..5462097 100644 --- a/crates/fold-ddb/src/mem.rs +++ b/crates/fold-ddb/src/mem.rs @@ -8,7 +8,7 @@ //! all — is measured on the same code path the DynamoDB table runs. use crate::layout::{typed_partition, ACTIVE, DATED, DATED_AT, MAX_ITEM_BYTES, PK, SK, TYPED}; -use crate::table::{Dated, Outcome, Row, Table}; +use crate::table::{Dated, Outcome, Row, Table, BATCH_GET_KEYS}; use crate::write::{attribute_bytes, beats, Cond, Write}; use nxs_foundation::change::Cell; use std::collections::BTreeMap; @@ -19,6 +19,9 @@ use std::sync::Mutex; #[derive(Debug, Clone, Copy, Default, PartialEq, Eq)] pub struct Requests { pub gets: u64, + /// `BatchGetItem` requests: one per [`BATCH_GET_KEYS`] distinct keys of a + /// [`get_many`](Table::get_many). + pub batch_gets: u64, pub writes: u64, pub deletes: u64, pub queries: u64, @@ -94,6 +97,16 @@ impl Table for MemTable { Ok(self.rows.lock().unwrap().get(sk).cloned()) } + async fn get_many(&self, sks: &[String]) -> Result, Infallible> { + let keys: std::collections::BTreeSet<&String> = sks.iter().collect(); + self.count(|r| r.batch_gets += keys.len().div_ceil(BATCH_GET_KEYS) as u64); + let rows = self.rows.lock().unwrap(); + Ok(keys + .into_iter() + .filter_map(|sk| rows.get(sk).map(|r| (sk.clone(), r.clone()))) + .collect()) + } + async fn write(&self, write: &Write) -> Result { self.count(|r| r.writes += 1); let mut rows = self.rows.lock().unwrap(); diff --git a/crates/fold-ddb/src/records.rs b/crates/fold-ddb/src/records.rs index 6560c95..bb0d35e 100644 --- a/crates/fold-ddb/src/records.rs +++ b/crates/fold-ddb/src/records.rs @@ -38,7 +38,8 @@ //! [`with_ranking`] fills the board's [`RankingGraph`] — the edges around the active tickets and //! their live neighbours — from the adjacency entries (index revision 3, 6j6v.1c5s), and the live //! items of a type whose `[ranking.list]` rank a key reads from the `typed` index (6j6v.xkr7). Its -//! cost is in its own doc. +//! cost is in its own doc. Where it reads many rows by key it sends `BatchGetItem`s +//! ([`Table::get_many`], 100 keys a request) instead of one `GetItem` each. //! //! # A hosted `list` //! @@ -64,11 +65,18 @@ //! (6j6v.15ed), the self-describing `.adj#` entries and the `typed` index attribute with revision 3 //! (6j6v.1c5s, 6j6v.xkr7). A stream whose watermark names an older one — or none, folded before //! revisions were recorded for the index — lacks them, and its records would come back without -//! labels or conversations, its list empty. [`active_board`] and [`list_board`] refuse such a -//! stream ([`RecordsError::NotCurrent`]); the fold run refolds it ([`Folder::is_current`]), after -//! which it reads normally. +//! labels or conversations, its list empty. [`active_board`], [`with_ranking`] and [`list_board`] +//! refuse such a stream ([`RecordsError::NotCurrent`]); the fold run refolds it +//! ([`Folder::is_current`]), after which it reads normally. +//! +//! A refold is refused for its whole length, not only until it starts: [`Folder::clear`] leaves a +//! sentinel watermark that names no revision (`NotCurrent`), and once the refold has written this +//! build's revisions the readers answer [`RecordsError::Refolding`] until it reaches the relay +//! position the clear named. A stream with no watermark at all was never folded, and reads as the +//! empty board it is. //! //! [`Folder::is_current`]: crate::fold::Folder::is_current +//! [`Folder::clear`]: crate::fold::Folder::clear use crate::board::{int, is_active, item_key, select, Selection, REMOVED}; use crate::fold::{Folder, INDEX_DOMAIN, INDEX_REVISION}; @@ -81,8 +89,8 @@ use futures_util::stream::{self, StreamExt, TryStreamExt}; use nexus_flow_core::graph::Edge; use nexus_flow_core::model::{EdgeKind, ItemRow, LinkWeight}; use nexus_flow_facade::read::{ - ActiveBoard, ActiveTicket, ListBoard, ListItem, ListReads, Neighbour, ParentRef, RankingGraph, - RankingReads, + self, ActiveBoard, ActiveTicket, ListBoard, ListItem, ListReads, ListSelect, Neighbour, + ParentRef, RankingGraph, RankingReads, }; use nxs_foundation::change::Cell; use std::collections::{BTreeMap, BTreeSet}; @@ -100,6 +108,10 @@ pub enum RecordsError { /// watermark names, `None` when it names none). Its records would lack joins; the fold run /// refolds it. NotCurrent { found: Option }, + /// The stream is being folded again from position 0 ([`Folder::clear`]) and has not yet + /// reached `until`, the relay position the refold has to reach before it is served: its + /// tables hold part of its history only. + Refolding { folded_through: i64, until: i64 }, } impl std::fmt::Display for RecordsError { @@ -111,6 +123,14 @@ impl std::fmt::Display for RecordsError { "the stream was folded by index revision {found:?}, this build reads revision \ {INDEX_REVISION}; it has to be folded again" ), + RecordsError::Refolding { + folded_through, + until, + } => write!( + f, + "the stream is being folded again, through relay position {folded_through} of \ + {until}; it is served once the refold reaches {until}" + ), } } } @@ -232,7 +252,8 @@ struct EdgeEntry { site: i64, } -/// The present edges touching ticket `id`, by tag: one Query on its `.adj#` prefix. +/// The present edges touching ticket `id`, by tag: one Query on its `.adj#` prefix. Only entries +/// that name `id` as an end count. async fn present_edges( table: &T, id: &str, @@ -250,6 +271,10 @@ async fn present_edges( }; Some((text(e, "tag")?.to_string(), entry)) }) + // An entry filed under `id` names it as an end. One that does not — an add whose tag was + // reused with other ends, leaving the entry under its old end behind — is not an edge of + // `id`, whatever its copy says (review of PR #55, Integrity #5). + .filter(|(_, e)| e.from == id || e.to == id) .collect()) } @@ -267,25 +292,32 @@ fn current_parent(id: &str, edges: &BTreeMap) -> Option#` | active ticket | /// | one Query on the `typed` index | type in `reads.list_types` (6j6v.xkr7) | /// | one Query on `.adj##` | live, unarchived item of such a type that is not active | -/// | one `GetItem` of its row | other end of such an edge that is not a subject | +/// | its row, by `BatchGetItem` | other end of such an edge that is not a subject — `⌈p / 100⌉` requests | /// | one Query on `.adj##` (its parent) | such end that is live | /// | one Query on `.ladj##`, `custom_fields##` | such end and listed item, when `reads` asks for labels, custom values | /// /// So `n` active tickets with `p` distinct inactive neighbours, `q` of them live, cost `n + q` -/// Queries and `p` `GetItem`s — `n + 3q` Queries with labels and custom values — at most +/// Queries, one `GetItem` and `⌈p / 100⌉` `BatchGetItem`s — `n + 3q` Queries with labels and +/// custom values — at most /// [`READ_CONCURRENCY`] items at a time, and nothing grows with an item's history: an entry says /// whether its add is removed. Every edge touching a subject is read once, from one end's entries, /// whatever the plugin's keys — the edge pass the store path runs once per query. A `next` key /// that reads the rank of a type only `[ranking.list]` orders adds, per such type, one index Query -/// and its live items as subjects ([`RankingReads::list_types`]). +/// and its live items as subjects ([`RankingReads::list_types`]) — one Query on `.adj#`, and one each +/// on `.ladj#` / `custom_fields#` when the keys read them, per live, unarchived item of the type, +/// however few the answer ranks: the rank is computed over the type's whole `list` scope +/// (docs/specs/derived-ranking-keys.md §12, decision 2). /// /// [`read::with_ranking`]: nexus_flow_facade::read::with_ranking /// [`read::next_ranking_reads`]: nexus_flow_facade::read::next_ranking_reads @@ -294,6 +326,9 @@ pub async fn with_ranking( mut board: ActiveBoard, reads: &RankingReads, ) -> Result> { + // The board may have been read before a refold began, and an older stream's entries say + // nothing of their edge: asked here again, one `GetItem` (review of PR #55, Integrity #3). + check_current(table).await?; let ids: Vec = board.tickets.keys().cloned().collect(); let held: BTreeMap = stream::iter(ids) .map(|id| async move { @@ -323,8 +358,15 @@ struct Held { edges: BTreeMap, } -/// The live, unarchived items of `item_type` on the `typed` index — the type's `list` scope — -/// asked of the row again, since the index may lag it. In id order. +/// The live, unarchived items of `item_type` on the `typed` index — the type's `list` scope — in +/// id order. +/// +/// What the index hands back is its copy of each row, and the copy's own cells are asked again: +/// the fold writes the `typed` attribute AFTER the row ([`board`](crate::board), "`typed`"), so a +/// copy taken between the two writes — or left by a run that died between them — can be retyped or +/// deleted while its attribute still names `item_type`; it is dropped. The base row itself is not +/// read again: the index's lag behind it is the one the module doc's "Two points in time" +/// describes, a state the stream really had, and is not hidden here. async fn typed_items(table: &T, item_type: &str) -> Result, T::Error> { Ok(table .query_typed(item_type) @@ -410,18 +452,13 @@ async fn ranking_graph( .filter(|id| !subject(id)) .cloned() .collect(); - let rows: Vec<(String, Option)> = stream::iter(others) - .map(|id| async move { - let row = table.get(&item_key(&id)).await?; - Ok::<_, T::Error>((id, row)) - }) - .buffered(READ_CONCURRENCY) - .try_collect() - .await?; - let live = rows + let keys: Vec = others.iter().map(|id| item_key(id)).collect(); + let mut rows = table.get_many(&keys).await?; + let live: Vec<(String, Row)> = others .into_iter() - .filter_map(|(id, row)| row.map(|r| (id, r))) - .filter(|(_, r)| text(r, "deleted") != Some("1")); + .filter_map(|id| rows.remove(&item_key(&id)).map(|r| (id, r))) + .filter(|(_, r)| text(r, "deleted") != Some("1")) + .collect(); let ends: BTreeMap = stream::iter(live) .map(|(id, row)| async move { let (n, _) = neighbour(table, &id, &row, reads).await?; @@ -477,36 +514,55 @@ async fn ranking_graph( /// [`read::list_reads`] for its query. The stream must be current, as for [`active_board`]: the /// `typed` index and the self-describing `.adj#` entries came with index revision 3. /// -/// | read | per | -/// |---------------------------------------------------------|----------------------------------| -/// | one `GetItem` of `.meta#watermark` | board | -/// | one Query on the `typed` index | listed type | -/// | one Query on the partition's `items#` rows | board without a type filter | -/// | one Query each on `.adj#`, `.ladj#`, `custom_fields#`; one `GetItem` of `item_timestamps` | listed item | -/// | one `GetItem` | parent of an open listed item outside the list | -/// | one Query on the `active` index, and per active ticket one Query each on `.ladj#` / `custom_fields#` when the graph reads them | board whose keys read a `next` rank | -/// | [`with_ranking`]'s reads around the listed items and active tickets | board with derived keys | +/// # What a list costs +/// +/// The listed types' rows come first: one Query on the `typed` index per listed type, or — for a +/// query without a type filter — one Query on the partition's `items#` rows (every item row of the +/// stream, deleted ones included, strongly consistent; DynamoDB pages it at 1 MB, so it is one +/// request per MB of item rows). Archived and deleted items are dropped there: `list` returns +/// neither. One `GetItem` of `.meta#watermark` per board. What follows depends on the order. +/// +/// **Any order but `rank`** ([`ListReads::select`]) reads the item rows alone, so the rows are +/// kept and cut BEFORE the joins a record carries are read ([`read::list_select`]): the status and +/// the type are asked of the row; a container filter reads one Query on `.adj#`, a label filter one +/// on `.ladj#`, per item the row kept; then only the `k` items of the page (at most `limit`) read +/// the rest: +/// +/// | read | per | +/// |--------------------------------------------------------------|---------------------------| +/// | one Query each on `.adj#`, `.ladj#`, `custom_fields#` (less what the filter read) | page item | +/// | `item_timestamps` rows, by `BatchGetItem` | page — `⌈k / 100⌉` requests | +/// | parents of open page items outside the page, by `BatchGetItem` | page — `⌈p / 100⌉` requests | /// -/// So `n` listed items cost `4n + 1 + p` requests and one Query per listed type, as -/// [`active_board`] costs per active ticket; the `.adj#` Query gives each item's parent, its -/// `contributes_to` targets and its parents for `parent_closed_reason` at once, and the graph reuses -/// it. Archived and deleted items are dropped before any join: `list` returns neither (archived -/// ones are on the index, live as they are, and filtered here as `list` filters them locally). +/// So `list --limit 10` costs `1 + 30 + 2` requests beside the rows' Query, at most, however many +/// items the stream holds and however many the status refuses — plus one Query per item the row +/// kept when the query filters by container or label. /// -/// A query without a type filter lists every live item, so it reads the partition's `items#` rows -/// — strongly consistent, every item row of the stream, deleted ones included — instead of an -/// index. The index lists one type; a query that names its types reads only those. +/// **`sort = rank`** reads every live, unarchived item of the listed types in full — one Query +/// each on `.adj#`, `.ladj#`, `custom_fields#` per item, and the timestamps and outside parents by +/// `BatchGetItem` — whatever its status, filter or limit: a rank is computed over the type's whole +/// `list` scope (docs/specs/derived-ranking-keys.md §12, decision 2), and a derived key reads the +/// labels and custom values of every item in it. When a key reads a rank `next` gives, one Query on +/// the `active` index and per active ticket one Query each on `.ladj#` / `custom_fields#` when the +/// graph reads them; with derived keys, [`with_ranking`]'s reads around the listed items and the +/// active tickets. The `.adj#` Query gives each item's parent, its `contributes_to` targets and its +/// parents for `parent_closed_reason` at once, and the graph reuses it. +/// +/// Nothing here caps the requests of one call: a ranked list of a type with `n` live items costs +/// about `3n` Queries, the number its rank is computed over. A server that must bound one request +/// sets its own budget around the table it hands in (`6j6v.wq9e`, E4 spec §5.1). /// /// [`read::list_board`]: nexus_flow_facade::read::list_board /// [`read::list_board_page`]: nexus_flow_facade::read::list_board_page /// [`read::list_board_value`]: nexus_flow_facade::read::list_board_value /// [`read::list_reads`]: nexus_flow_facade::read::list_reads +/// [`read::list_select`]: nexus_flow_facade::read::list_select pub async fn list_board( table: &T, reads: &ListReads, ) -> Result> { check_current(table).await?; - let rows: Vec = match &reads.types { + let mut rows: Vec = match &reads.types { Some(types) => { let mut rows = Vec::new(); for ty in types { @@ -525,27 +581,88 @@ pub async fn list_board( }) .collect(), }; - let read: Vec<(Row, BTreeMap, ListItem)> = stream::iter(rows) + let keep = reads.select.as_ref(); + // A list that is not ranked keeps what its row decides — status, type — before any join. + if let Some(select) = keep { + rows.retain(|r| select.keeps_row(&item_row(r, None))); + } + + // What keeping an item reads: for a ranked list every join below, for any other only what its + // filter reads — a container's edges, a label's labels. + let edges_first = keep.is_none_or(ListSelect::reads_edges); + let labels_first = keep.is_none_or(ListSelect::reads_labels); + type Read = (Row, Option>, ListItem); + let first: Vec = stream::iter(rows) .map(|row| async move { let id = text(&row, "id").unwrap_or_default().to_string(); - let edges = present_edges(table, &id).await?; - let (created_at, updated_at) = timestamps(table, &id).await?; - let mut i = ListItem::new(item_row(&row, current_parent(&id, &edges))); - i.labels = labels(table, &id).await?; - i.custom = custom(table, &id).await?; - i.created_at = created_at; - i.updated_at = updated_at; - let targets: BTreeSet = edges - .values() - .filter(|e| e.kind == EdgeKind::ContributesTo && e.from == id) - .map(|e| e.to.clone()) + let edges = if edges_first { + Some(present_edges(table, &id).await?) + } else { + None + }; + let mut i = list_item(&row, &id, edges.as_ref()); + if labels_first { + i.labels = labels(table, &id).await?; + } + Ok::<_, T::Error>((row, edges, i)) + }) + .buffered(READ_CONCURRENCY) + .try_collect() + .await?; + + // Kept and cut as `list_board_page` keeps and cuts, so the joins below are read for the page. + let (first, cut) = match keep { + None => (first, None), + Some(select) => { + let mut held: BTreeMap>)> = + BTreeMap::new(); + let items: Vec = first + .into_iter() + .map(|(row, edges, i)| { + held.insert(i.item.id.clone(), (row, edges)); + i + }) .collect(); - i.contributes_to = targets.into_iter().collect(); + let (kept, cut) = read::list_select(select, items); + let kept = kept + .into_iter() + .filter_map(|i| held.remove(&i.item.id).map(|(row, edges)| (row, edges, i))) + .collect(); + (kept, Some(cut)) + } + }; + + // The rest of each kept item's joins. + let mut read: Vec<(Row, BTreeMap, ListItem)> = stream::iter(first) + .map(|(row, edges, mut i)| async move { + let id = i.item.id.clone(); + let edges = match edges { + Some(edges) => edges, + None => { + let edges = present_edges(table, &id).await?; + let full = list_item(&row, &id, Some(&edges)); + i.item = full.item; + i.contributes_to = full.contributes_to; + edges + } + }; + if !labels_first { + i.labels = labels(table, &id).await?; + } + i.custom = custom(table, &id).await?; Ok::<_, T::Error>((row, edges, i)) }) .buffered(READ_CONCURRENCY) .try_collect() .await?; + let keys: Vec = read + .iter() + .map(|(_, _, i)| timestamps_key(&i.item.id)) + .collect(); + let stamped = table.get_many(&keys).await?; + for (_, _, i) in &mut read { + (i.created_at, i.updated_at) = stamps(stamped.get(×tamps_key(&i.item.id))); + } // `parent_closed_reason`: an OPEN item with parents, none of them active, lists them all. A // parent outside the list is read once, however many children it has. @@ -560,25 +677,19 @@ pub async fn list_board( .iter() .map(|(row, _, i)| (i.item.id.as_str(), row)) .collect(); - let outside: BTreeSet = read + let outside: Vec = read .iter() .filter(|(_, _, i)| i.item.status.as_deref() == Some("open")) .flat_map(|(_, edges, i)| parents_of(&i.item.id, edges)) .filter(|p| !listed_rows.contains_key(p.as_str())) + .map(|p| item_key(&p)) .collect(); - let outside: BTreeMap> = stream::iter(outside) - .map(|pid| async move { - let row = table.get(&item_key(&pid)).await?; - Ok::<_, T::Error>((pid, row)) - }) - .buffered(READ_CONCURRENCY) - .try_collect() - .await?; + let outside = table.get_many(&outside).await?; let parent_row = |pid: &str| -> Option<&Row> { listed_rows .get(pid) .copied() - .or_else(|| outside.get(pid).and_then(Option::as_ref)) + .or_else(|| outside.get(&item_key(pid))) }; let parent_closed: Vec)>> = read .iter() @@ -615,6 +726,7 @@ pub async fn list_board( items.push(i); } let mut board = ListBoard::new(reads.types.clone(), items); + board.cut = cut; let graph_reads = reads.ranking.clone().unwrap_or_default(); if reads.next { @@ -666,13 +778,38 @@ pub async fn list_board( Ok(board) } -/// Refuse a stream folded by older index rules than this build reads. +/// A listed item from its row and, when they were read, its present edges: its `belongs_to` and its +/// `contributes_to` targets come from them, and stay empty without them. +fn list_item(row: &Row, id: &str, edges: Option<&BTreeMap>) -> ListItem { + let mut i = ListItem::new(item_row(row, edges.and_then(|e| current_parent(id, e)))); + if let Some(edges) = edges { + let targets: BTreeSet = edges + .values() + .filter(|e| e.kind == EdgeKind::ContributesTo && e.from == id) + .map(|e| e.to.clone()) + .collect(); + i.contributes_to = targets.into_iter().collect(); + } + i +} + +/// Refuse a stream folded by older index rules than this build reads, and one being folded again +/// that has not caught up yet. A cleared stream's watermark names no revision +/// ([`Folder::clear`]), so it is refused from the clear on; once the refold has written this +/// build's revisions it is still refused until it reaches the position the clear named. A stream +/// with no watermark at all has never been folded: it reads as the empty board it is. async fn check_current(table: &T) -> Result<(), RecordsError> { if let Some(mark) = Folder::watermark(table).await? { let found = mark.revisions.get(INDEX_DOMAIN).copied(); if found != Some(INDEX_REVISION) { return Err(RecordsError::NotCurrent { found }); } + if let Some(until) = mark.refold_until.filter(|u| mark.folded_through < *u) { + return Err(RecordsError::Refolding { + folded_through: mark.folded_through, + until, + }); + } } Ok(()) } @@ -714,16 +851,23 @@ async fn timestamps( table: &T, id: &str, ) -> Result<(Option, Option), T::Error> { - let row = table - .get(&key("item_timestamps", &[("item_id", id)])) - .await?; + let row = table.get(×tamps_key(id)).await?; + Ok(stamps(row.as_ref())) +} + +/// The sort key of the ticket's `item_timestamps` row. +fn timestamps_key(id: &str) -> String { + key("item_timestamps", &[("item_id", id)]) +} + +/// The creation and last change an `item_timestamps` row holds, `''` read as none. +fn stamps(row: Option<&Row>) -> (Option, Option) { let stamp = |c: &str| { - row.as_ref() - .and_then(|r| text(r, c)) + row.and_then(|r| text(r, c)) .filter(|v| !v.is_empty()) .map(str::to_string) }; - Ok((stamp("created_at"), stamp("updated_at"))) + (stamp("created_at"), stamp("updated_at")) } /// Every non-empty custom value of the ticket, by field. diff --git a/crates/fold-ddb/src/snapshot.rs b/crates/fold-ddb/src/snapshot.rs index 55b90a7..e3c9091 100644 --- a/crates/fold-ddb/src/snapshot.rs +++ b/crates/fold-ddb/src/snapshot.rs @@ -7,9 +7,11 @@ //! change is idempotent. The order is what makes that safe: tables read before the watermark could //! lack an op the watermark claims. //! -//! [`import`] starts an EMPTY stream from one: it writes the rows, rebuilds the board's indexes from -//! them, and writes the watermark last, so a stream whose import died has no watermark and is -//! started again. The log stays where it is — on the relay — and the run then folds it from the +//! [`import`] starts an EMPTY stream from one — one never folded, or one [`Folder::clear`] left — +//! and keeps it unserved while it does: it writes a sentinel watermark that names no revision +//! first, so readers answer `NotCurrent` throughout; then the rows; it rebuilds the board's +//! indexes from them, and writes the real watermark last, so a stream whose import died is still +//! unserved and is started again. The log stays where it is — on the relay — and the run then folds it from the //! watermark on, in batches. //! //! [`from_sqlite`] makes one from a local replica's views, for a server that starts from a board @@ -186,10 +188,12 @@ pub async fn import( folder: folder.revisions(), }); } + // A cleared stream's watermark — and that of an import that died — names no revision: it + // holds nothing to keep. if Folder::watermark(table) .await .map_err(ImportError::Table)? - .is_some() + .is_some_and(|mark| !mark.revisions.is_empty()) { return Err(ImportError::NotEmpty); } @@ -232,18 +236,23 @@ pub async fn import( }); } } + // Unserved from the first row on: a watermark without revisions (review of PR #55, + // Integrity #2). + table + .write(&Folder::sentinel(0, None)) + .await + .map_err(ImportError::Table)?; for write in &writes { table.write(write).await.map_err(ImportError::Table)?; } + // An item whose row has no room for its index attributes keeps none: as with the row writes + // above, whose `TooLarge` an import does not report either (`6j6v.jtfy`). board::reindex(table).await.map_err(ImportError::Table)?; - // A NEW watermark, not an advance: the stream had none. - folder - .advance(table, snapshot.folded_through) + // The real watermark over the sentinel, unconditionally: the stream had none of its own. + table + .write(&folder.stamp(snapshot.folded_through)) .await - .map_err(|e| match e { - crate::fold::FoldError::Table(e) => ImportError::Table(e), - crate::fold::FoldError::Layout(e) => ImportError::Layout(e), - })?; + .map_err(ImportError::Table)?; Ok(()) } @@ -313,5 +322,6 @@ pub fn watermark(snapshot: &Snapshot) -> Watermark { Watermark { folded_through: snapshot.folded_through, revisions: snapshot.revisions.clone(), + refold_until: None, } } diff --git a/crates/fold-ddb/src/table.rs b/crates/fold-ddb/src/table.rs index 51cd92b..b60ffab 100644 --- a/crates/fold-ddb/src/table.rs +++ b/crates/fold-ddb/src/table.rs @@ -10,6 +10,9 @@ use std::future::Future; /// A stored row: its attributes by name, the sort key and index attributes among them. pub type Row = BTreeMap; +/// The most keys one `BatchGetItem` names — DynamoDB's own limit ([`Table::get_many`]). +pub const BATCH_GET_KEYS: usize = 100; + /// What a write did. #[derive(Debug, Clone, Copy, PartialEq, Eq)] pub enum Outcome { @@ -49,6 +52,16 @@ pub trait Table: Sync { /// One row, read strongly consistent. fn get(&self, sk: &str) -> impl Future, Self::Error>> + Send; + /// Several rows, read strongly consistent, by sort key — a key without a row is absent from the + /// answer, and a key named twice is read once. DynamoDB answers it with `BatchGetItem`, at most + /// [`BATCH_GET_KEYS`] keys a request, sending the keys a request left unprocessed again; so it + /// costs `⌈keys / 100⌉` requests, more only when DynamoDB throttles or a response passes 16 MB. + /// No keys, no request. + fn get_many( + &self, + sks: &[String], + ) -> impl Future, Self::Error>> + Send; + /// One conditional write. fn write(&self, write: &Write) -> impl Future> + Send; diff --git a/crates/fold-ddb/tests/common/mod.rs b/crates/fold-ddb/tests/common/mod.rs index 8d4274c..55094dc 100644 --- a/crates/fold-ddb/tests/common/mod.rs +++ b/crates/fold-ddb/tests/common/mod.rs @@ -185,6 +185,12 @@ pub fn delivered(s: &Store, seed: u64) -> Vec<(i64, Op)> { .collect() } +/// The highest relay position among `ops` — the head a refold of them has to reach +/// (`Folder::clear`). +pub fn head(ops: &[(i64, Op)]) -> i64 { + ops.iter().map(|(at, _)| *at).max().unwrap_or(0) +} + /// Every row of `table`, every column filled — as SQLite holds it. pub fn sqlite_rows(s: &Store, table: &str) -> Vec> { let conn = s.connection(); diff --git a/crates/fold-ddb/tests/derived_differential.rs b/crates/fold-ddb/tests/derived_differential.rs index cd5b228..8beb9c9 100644 --- a/crates/fold-ddb/tests/derived_differential.rs +++ b/crates/fold-ddb/tests/derived_differential.rs @@ -26,7 +26,7 @@ use nexus_flow_core::model::{EdgeKind, ItemRow, MergeStrategy}; use nexus_flow_core::store::Store; use nexus_flow_facade::plugin::{self, PluginConfig, PluginRegistration}; use nexus_flow_facade::read; -use nxs_fold_ddb::fold::{Folder, INDEX_DOMAIN}; +use nxs_fold_ddb::fold::{Folder, INDEX_DOMAIN, INDEX_REVISION}; use nxs_fold_ddb::layout::WATERMARK_KEY; use nxs_fold_ddb::mem::{ready, MemTable}; use nxs_fold_ddb::table::Table; @@ -36,7 +36,7 @@ use nxs_foundation::change::Cell; use std::collections::{BTreeMap, BTreeSet}; mod common; -use common::{delivered, Lcg, NOW}; +use common::{delivered, head, Lcg, NOW}; /// Every derived key kind, each able to read a neighbour that is not active: a goal's order and /// rank from a closed or archived goal, counts without a status filter, a `dep` blocker's status @@ -213,6 +213,10 @@ fn ids(items: &[ItemRow]) -> Vec { items.iter().map(|i| i.id.clone()).collect() } +/// An active ticket, as the SQL `items` row `a` says: live, unarchived, open or in progress. +const ACTIVE_SQL: &str = "COALESCE(a.deleted, '') != '1' AND a.archived IS NULL + AND a.status IN ('open', 'in_progress')"; + fn count(s: &Store, sql: &str) -> usize { s.connection() .query_row(sql, [], |r| r.get::<_, i64>(0)) @@ -326,12 +330,18 @@ fn the_server_ranks_next_by_derived_keys_as_the_replica_does() { ), ); } + // Counted where they matter (review of PR #55, Test #4): at an active ticket, whose edges + // the graph reads — not anywhere on the board. seen.add( "edges dropped for a deleted end", count( &s, - "SELECT COUNT(*) FROM present_edges e JOIN items i - ON i.id IN (e.from_id, e.to_id) WHERE i.deleted = '1'", + &format!( + "SELECT COUNT(*) FROM present_edges e + JOIN items d ON d.id IN (e.from_id, e.to_id) AND d.deleted = '1' + JOIN items a ON a.id IN (e.from_id, e.to_id) AND a.id != d.id + WHERE {ACTIVE_SQL}" + ), ), ); // A child with two present parent edges: only `present_parent`'s pick may count. @@ -344,8 +354,14 @@ fn the_server_ranks_next_by_derived_keys_as_the_replica_does() { ), ); seen.add( - "removed edges", - count(&s, "SELECT COUNT(*) FROM edge_removes"), + "removed edges at an active ticket", + count( + &s, + &format!( + "SELECT COUNT(*) FROM edge_removes r JOIN edge_adds e ON e.tag = r.tag + JOIN items a ON a.id IN (e.from_id, e.to_id) WHERE {ACTIVE_SQL}" + ), + ), ); // A derived key decides: a project ranked ahead of one with a better priority. let projects: Vec<&ItemRow> = want @@ -369,6 +385,7 @@ fn the_server_ranks_next_by_derived_keys_as_the_replica_does() { "inactive children of a candidate", "dep edges from a candidate to an inactive blocker", "edges dropped for a deleted end", + "removed edges at an active ticket", "children parented twice", "projects ordered against their priority", ] { @@ -409,8 +426,10 @@ fn reading_the_ranking_graph_costs_one_query_per_ticket_and_per_live_neighbour() let after = t.requests(); // One adjacency Query per active ticket; the live neighbour's parent, labels and custom values. assert_eq!(after.queries - before.queries, n + 3); - // One row per inactive neighbour: the closed goal and the deleted one. - assert_eq!(after.gets - before.gets, 2); + // The watermark, asked again (review of PR #55, Integrity #3); the rows of the inactive + // neighbours — the closed goal and the deleted one — in one `BatchGetItem`. + assert_eq!(after.gets - before.gets, 1); + assert_eq!(after.batch_gets - before.batch_gets, 1); assert_eq!(after.index_queries, before.index_queries); assert_eq!(after.writes, before.writes); let local = read::with_ranking(&s, read::active_board(&s).unwrap(), &reads).unwrap(); @@ -440,6 +459,8 @@ fn a_stream_folded_by_index_revision_2_is_refused_then_refolded_and_ranks() { let s = gen_board(seed); let ops = delivered(&s, seed); let t = fold(&folder, &ops); + // A board read while the stream was current — a server that holds one across the refold. + let plain = ready(records::active_board(&t, ready(board::select(&t)).unwrap())).unwrap(); for (sk, row) in t.rows() { if let Some(Cell::Text(tag)) = row.get("tag").filter(|_| sk.starts_with(".adj#")) { ready( @@ -468,9 +489,33 @@ fn a_stream_folded_by_index_revision_2_is_refused_then_refolded_and_ranks() { Err(records::RecordsError::NotCurrent { found: Some(2) }) => {} other => panic!("seed {seed}: a revision-2 stream must be refused, got {other:?}"), } + // The graph read refuses it itself (review of PR #55, Integrity #3)… + match ready(records::with_ranking(&t, plain.clone(), &reads)) { + Err(records::RecordsError::NotCurrent { found: Some(2) }) => {} + other => panic!( + "seed {seed}: the graph of a revision-2 stream must be refused, got {other:?}" + ), + } + // …because the stripped entries would read as no edge at all: under a watermark that + // claims revision 3, the same read answers an empty graph where the store has edges. + let local = read::with_ranking(&s, read::active_board(&s).unwrap(), &reads).unwrap(); + let claim = |revision: i64| { + ready(t.write(&Write { + sk: WATERMARK_KEY.into(), + set: vec![(format!("revision_{INDEX_DOMAIN}"), Cell::Int(revision))], + remove: Vec::new(), + condition: Some(Cond::RowPresent), + })) + .unwrap(); + }; + claim(INDEX_REVISION); + let stripped = ready(records::with_ranking(&t, plain.clone(), &reads)).unwrap(); + assert!(stripped.ranking.unwrap().edges.is_empty(), "seed {seed}"); + assert!(!local.ranking.unwrap().edges.is_empty(), "seed {seed}"); + claim(2); let mark = ready(Folder::watermark(&t)).unwrap().unwrap(); assert!(!folder.is_current(&mark), "seed {seed}"); - ready(Folder::clear(&t)).unwrap(); + ready(Folder::clear(&t, head(&ops))).unwrap(); ready(folder.fold_batch(&t, &ops)).unwrap(); let server = served(&t, &reads); diff --git a/crates/fold-ddb/tests/dynamodb_local.rs b/crates/fold-ddb/tests/dynamodb_local.rs index d664758..7292c4f 100644 --- a/crates/fold-ddb/tests/dynamodb_local.rs +++ b/crates/fold-ddb/tests/dynamodb_local.rs @@ -1,6 +1,8 @@ //! The DynamoDB adapter against a real DynamoDB Local (6j6v.k7w7): the same fold written into //! DynamoDB and into a [`MemTable`] leaves the same items, the board reads back through the indexes -//! with the same lanes and the same `typed` listings (6j6v.xkr7), and every request the client sends is recorded — none is a Scan. +//! with the same lanes and the same `typed` listings (6j6v.xkr7), the ranked list and the ranking +//! graph read back as from memory — a hub whose adjacency spans Query pages among them (review of +//! PR #55) — and every request the client sends is recorded — none is a Scan. //! //! Gated like `nxs-server`'s suite: on the `dynamodb` feature and `NXF_RELAY_DDB_ENDPOINT`, skipped //! locally without one and a hard failure under CI. Each test creates its own table from @@ -181,6 +183,11 @@ async fn a_board_folded_into_dynamodb_reads_back_as_from_memory_and_no_request_i counted.queries + counted.index_queries, "{sent:?}" ); + assert_eq!( + sent.get("BatchGetItem").copied().unwrap_or(0), + counted.batch_gets, + "{sent:?}" + ); let want = board::select(&mem).await.unwrap().board.lanes(NOW); let ddb_ref = &ddb; @@ -271,11 +278,12 @@ async fn closing_a_ticket_with_many_dependents_costs_one_query_and_a_few_request let sent = recorder.take(); eprintln!("closing a ticket with {n} dependents sent {sent:?}"); let count = |op: &str| sent.get(op).copied().unwrap_or(0); - // The close: three changes and the watermark. The ticket's refresh: one read, one write. The - // flip: one adjacency Query, and per edge three reads — the edge, its remove, its first end, - // a dependent that is still active — and no write, since that keeps the edge on the index. + // The close: three changes and the watermark, one read to weigh the ticket's row first. The + // ticket's refresh: one read, one write. The flip: one adjacency Query, and per edge three + // reads — the edge, its remove, its first end, a dependent that is still active — and no + // write, since that keeps the edge on the index. assert_eq!(count("Query"), 1, "{sent:?}"); - assert_eq!(count("GetItem"), 1 + 3 * n, "{sent:?}"); + assert_eq!(count("GetItem"), 2 + 3 * n, "{sent:?}"); assert_eq!(count("UpdateItem"), 3 + 1 + 1, "{sent:?}"); assert_eq!(count("Scan"), 0); @@ -452,3 +460,162 @@ async fn the_records_read_back_as_from_memory_with_four_requests_per_ticket_and_ ); } } + +/// The reads of a ranked list of `types` (`None`: every type) whose keys follow every edge kind, +/// read labels and custom values, the `next` board and the `list` scope of `list_types`. +fn ranked_reads(types: Option<&[&str]>, list_types: &[&str]) -> nexus_flow_facade::read::ListReads { + let mut graph = nexus_flow_facade::read::RankingReads::default(); + graph.kinds = [ + EdgeKind::Parent, + EdgeKind::ContributesTo, + EdgeKind::Mentions, + EdgeKind::Dep, + ] + .into_iter() + .collect(); + graph.labels = true; + graph.custom = true; + graph.list_types = list_types.iter().map(|t| t.to_string()).collect(); + let mut reads = nexus_flow_facade::read::ListReads::default(); + reads.types = types.map(|t| t.iter().map(|t| t.to_string()).collect()); + reads.next = true; + reads.ranking = Some(graph); + reads +} + +/// What a list board holds, comparably. +type ListView = ( + BTreeMap, + Option, + Option>, +); + +async fn list_view(t: &T, reads: &nexus_flow_facade::read::ListReads) -> ListView +where + T::Error: std::fmt::Debug, +{ + let board = records::list_board(t, reads).await.unwrap(); + (board.items, board.ranking, board.active.map(|a| a.tickets)) +} + +#[tokio::test(flavor = "multi_thread")] +async fn the_ranked_list_and_the_ranking_graph_read_back_as_from_memory() { + // Review of PR #55, Integrity #7: `records::list_board` and `records::with_ranking` through the + // SDK — the `typed` Query joined with base-table reads, `BatchGetItem` — give what they give + // over the memory table, and send no Scan. + let Some(url) = endpoint("ranked") else { + return; + }; + let recorder = Recorder::default(); + let client = client(&url, &recorder); + let name = fresh_table(&client, "ranked").await; + let folder = Folder::platform().unwrap(); + for seed in [3, 19, 41] { + let s = merged(seed); + let ops = delivered(&s, seed); + let stream = format!("stream-ranked-{seed}"); + let ddb = DynamoDbTable::new(client.clone(), &name, &stream); + let mem = MemTable::new(&stream); + folder.fold_batch(&ddb, &ops).await.unwrap(); + folder.fold_batch(&mem, &ops).await.unwrap(); + let mut all = vec![ranked_reads(None, &[])]; + for ty in TYPES { + let others: Vec<&str> = TYPES.into_iter().filter(|t| *t != ty).collect(); + all.push(ranked_reads(Some(&[ty]), &others)); + } + for reads in &all { + let want = list_view(&mem, reads).await; + let ddb_ref = &ddb; + let got = eventually(&want, || async move { list_view(ddb_ref, reads).await }).await; + assert_eq!(got, want, "seed {seed}, {:?}", reads.types); + } + let graph = ranked_reads(None, &["epic"]).ranking.unwrap(); + let plain = records::active_board(&mem, board::select(&mem).await.unwrap()) + .await + .unwrap(); + let want = records::with_ranking(&mem, plain.clone(), &graph) + .await + .unwrap() + .ranking; + let ddb_ref = &ddb; + let (plain_ref, graph_ref) = (&plain, &graph); + let got = eventually(&want, || async move { + records::with_ranking(ddb_ref, plain_ref.clone(), graph_ref) + .await + .unwrap() + .ranking + }) + .await; + assert_eq!(got, want, "seed {seed}: the ranking graph"); + let sent = recorder.take(); + assert!(sent.contains_key("BatchGetItem"), "{sent:?}"); + assert!(!sent.contains_key("Scan"), "{sent:?}"); + } +} + +#[tokio::test(flavor = "multi_thread")] +async fn a_hub_whose_adjacency_spans_query_pages_reads_every_edge() { + // Review of PR #55, Integrity #7: an item with more adjacency entries than one Query page holds + // (1 MB) — 450 contributors with ids near the key limit, some 2.9 KB an entry — reads every edge + // through the SDK, as from memory. + let Some(url) = endpoint("hub") else { return }; + let recorder = Recorder::default(); + let client = client(&url, &recorder); + let name = fresh_table(&client, "hub").await; + let folder = Folder::platform().unwrap(); + let hub = format!("ab12.h{}", "h".repeat(890)); + let mut s = Store::open_in_memory(1); + s.create_item(&hub, "epic", "Hub", "u"); + let n = 450; + for i in 0..n { + let id = format!("ab12.{i:04}{}", "n".repeat(890)); + s.create_item(&id, "task", "Contributor", "u"); + s.add_edge(&id, &hub, EdgeKind::ContributesTo, "u"); + } + let ops: Vec<(i64, _)> = s + .export() + .into_iter() + .enumerate() + .map(|(i, op)| (i as i64 + 1, op)) + .collect(); + let ddb = DynamoDbTable::new(client.clone(), &name, "stream-hub"); + let mem = MemTable::new("stream-hub"); + assert!(folder.fold_batch(&ddb, &ops).await.unwrap().is_empty()); + folder.fold_batch(&mem, &ops).await.unwrap(); + + recorder.take(); + let entries = ddb.query_prefix(&format!(".adj#{hub}#")).await.unwrap(); + let sent = recorder.take(); + assert_eq!(entries.len(), n); + assert!( + sent.get("Query").copied().unwrap_or(0) >= 2, + "the hub's entries span pages: {sent:?}" + ); + + let mut graph = nexus_flow_facade::read::RankingReads::default(); + graph.kinds.insert(EdgeKind::ContributesTo); + let plain = records::active_board(&mem, board::select(&mem).await.unwrap()) + .await + .unwrap(); + let want = records::with_ranking(&mem, plain.clone(), &graph) + .await + .unwrap() + .ranking + .unwrap(); + assert_eq!(want.edges.len(), n); + let got = records::with_ranking(&ddb, plain, &graph) + .await + .unwrap() + .ranking + .unwrap(); + assert_eq!(got, want); + + // The hub listed by its type: its edges come from the same paged entries. + let reads = ranked_reads(Some(&["epic"]), &[]); + let want = list_view(&mem, &reads).await; + let ddb_ref = &ddb; + let reads_ref = &reads; + let got = eventually(&want, || async move { list_view(ddb_ref, reads_ref).await }).await; + assert_eq!(got, want); + assert_eq!(got.1.unwrap().edges.len(), n); +} diff --git a/crates/fold-ddb/tests/fold_differential.rs b/crates/fold-ddb/tests/fold_differential.rs index 824caa8..d887b13 100644 --- a/crates/fold-ddb/tests/fold_differential.rs +++ b/crates/fold-ddb/tests/fold_differential.rs @@ -14,6 +14,7 @@ use nexus_flow_core::model::EdgeKind; use nexus_flow_core::store::Store; use nexus_flow_core::task_reducer::TaskReducer; use nxs_fold_ddb::fold::Folder; +use nxs_fold_ddb::layout::WATERMARK_KEY; use nxs_fold_ddb::mem::{ready, MemTable}; use nxs_fold_ddb::table::{text, Dated, Table}; use nxs_fold_ddb::{board, chunks}; @@ -273,8 +274,14 @@ fn the_watermark_only_moves_forward() { ); let rows: BTreeMap<_, _> = t.rows(); assert_eq!(rows.len(), 1); - ready(Folder::clear(&t)).unwrap(); - assert!(t.rows().is_empty()); + // A clear leaves the watermark only, as a sentinel: no revision, so no reader serves the + // stream, and the position its refold has to reach. + ready(Folder::clear(&t, 10)).unwrap(); + assert_eq!(t.rows().keys().collect::>(), [WATERMARK_KEY]); + let mark = ready(Folder::watermark(&t)).unwrap().unwrap(); + assert!(mark.revisions.is_empty()); + assert_eq!((mark.folded_through, mark.refold_until), (0, Some(10))); + assert!(!folder.is_current(&mark)); } #[test] @@ -303,10 +310,12 @@ fn a_flip_costs_one_query_and_three_reads_per_edge_and_no_more() { let before = t.requests(); ready(folder.fold_batch(&t, &all(&s, seen))).unwrap(); let after = t.requests(); - // Three changes, the ticket's index write, the watermark; one read for the ticket and three - // per edge; one adjacency Query. Nothing grows with the board beyond the closed ticket's edges. + // Three changes, the ticket's index write, the watermark; two reads for the ticket — one to + // weigh its row before the op writes (review of PR #55, Integrity #4), one for its index — and + // three per edge; one adjacency Query. Nothing grows with the board beyond the closed ticket's + // edges. assert_eq!(after.writes - before.writes, 5); - assert_eq!(after.gets - before.gets, 1 + 3 * n); + assert_eq!(after.gets - before.gets, 2 + 3 * n); assert_eq!(after.queries - before.queries, 1); assert_eq!( ready(board::select(&t)) diff --git a/crates/fold-ddb/tests/list_differential.rs b/crates/fold-ddb/tests/list_differential.rs index c44e6fd..af8a713 100644 --- a/crates/fold-ddb/tests/list_differential.rs +++ b/crates/fold-ddb/tests/list_differential.rs @@ -228,6 +228,15 @@ fn queries(container: &str) -> Vec { ranked().status("closed").filter(types(&["person"])), ListQuery::new().filter(types(&["person"])), ranked().filter(types(&["person"])).limit(3), + // Not ranked: kept and cut before the joins (review of PR #55, Integrity #1). + ListQuery::new().status("open").limit(4), + ListQuery::new() + .sort(SortKey::Closed) + .filter(NextFilter::new().with_label("waiting")) + .limit(2), + ListQuery::new() + .filter(types(&["action", "project"]).with_container(container)) + .limit(1), ] } @@ -295,11 +304,58 @@ fn the_server_answers_a_ranked_list_as_the_replica_does() { if let Some(graph) = &server.ranking { seen.add("graph listed items", graph.listed.len()); seen.add("graph edges", graph.edges.len()); + // What a list graph must hold and the lanes would not (review of PR #55, Test #5): + // closed and archived neighbours, inbound `contributes_to` into a listed item, and + // archived items of a graph list type, which its scope leaves out. + for n in graph.neighbours.values() { + let archived = n.item.archived.is_some(); + seen.add( + "closed neighbours in a list graph", + usize::from(!archived && n.item.status.as_deref() == Some("closed")), + ); + seen.add("archived neighbours in a list graph", usize::from(archived)); + } + seen.add( + "inbound contributes_to into a listed item", + graph + .edges + .iter() + .filter(|e| { + e.kind == EdgeKind::ContributesTo && server.items.contains_key(&e.to) + }) + .count(), + ); + let archived_in_scope: Vec = s + .list_items() + .unwrap() + .into_iter() + .filter(|i| i.archived.is_some() && i.deleted.as_deref() != Some("1")) + .filter(|i| { + i.item_type + .as_deref() + .is_some_and(|t| graph.reads.list_types.contains(t)) + }) + .map(|i| i.id) + .collect(); + assert!( + archived_in_scope + .iter() + .all(|id| !graph.listed.contains(id)), + "an archived item is out of the list scope, {ctx}" + ); + seen.add( + "archived items of a graph list type", + archived_in_scope.len(), + ); } seen.add( "boards with next's board", usize::from(server.active.is_some()), ); + seen.add( + "boards kept and cut before their joins", + usize::from(server.cut.is_some()), + ); seen.add( "listed items with a closed-parent reason", server @@ -360,6 +416,11 @@ fn the_server_answers_a_ranked_list_as_the_replica_does() { "closed persons listed", "next rows", "archived items", + "closed neighbours in a list graph", + "archived neighbours in a list graph", + "inbound contributes_to into a listed item", + "archived items of a graph list type", + "boards kept and cut before their joins", ] { assert!( seen.0.get(what).copied().unwrap_or(0) > 0, @@ -398,10 +459,11 @@ fn a_stream_not_folded_by_this_index_revision_is_refused_for_a_list() { } #[test] -fn reading_a_typed_list_costs_one_index_query_and_four_requests_per_item() { +fn reading_a_typed_list_costs_one_index_query_three_per_item_and_one_batch() { // §9 "bounded cost": the persons of a board are one Query on the `typed` index, and each costs - // its adjacency, labels, custom values and timestamps — whatever else the board holds, which - // this list reads nothing of. A type filter reads only its own type. + // its adjacency, labels and custom values, the timestamps of all of them one `BatchGetItem` — + // whatever else the board holds, which this list reads nothing of. A type filter reads only its + // own type. let folder = Folder::platform().unwrap(); let cfg = cfg(); let mut s = Store::open_in_memory(1); @@ -425,8 +487,9 @@ fn reading_a_typed_list_costs_one_index_query_and_four_requests_per_item() { let after = t.requests(); assert_eq!(after.index_queries - before.index_queries, 1); assert_eq!(after.queries - before.queries, 3 * n); - // The watermark, and each person's timestamps; no parents. - assert_eq!(after.gets - before.gets, 1 + n); + // The watermark; the persons' timestamps in one batch; no parents. + assert_eq!(after.gets - before.gets, 1); + assert_eq!(after.batch_gets - before.batch_gets, 1); assert_eq!(after.writes, before.writes); assert_eq!( server.items.len(), @@ -439,3 +502,188 @@ fn reading_a_typed_list_costs_one_index_query_and_four_requests_per_item() { ids(&read::list_page(&cfg, &s, &q).unwrap().items) ); } + +#[test] +fn an_unranked_list_costs_its_page_not_the_stream() { + // Review of PR #55, Integrity #1: `list --status open --limit 5` over a stream of many closed + // items reads the joins of its five rows, not of every item. The rows' own Query is the one + // read that grows with the stream; every other request is the same with 0 closed items or 300. + let folder = Folder::platform().unwrap(); + let cfg = cfg(); + let cost = |closed: usize| { + let mut s = Store::open_in_memory(1); + for i in 0..12 { + let id = format!("ab12.a{i:03}"); + s.create_item(&id, "action", "Open", "u"); + if i % 3 == 0 { + s.add_label(&id, "waiting", "u"); + } + } + for i in 0..closed { + let id = format!("ab12.c{i:03}"); + s.create_item(&id, "action", "Closed", "u"); + s.set_field(&id, "status", Some("closed".into()), "u"); + } + let t = MemTable::new("stream-1"); + ready(folder.fold_batch(&t, &delivered(&s, 7))).unwrap(); + let mut costs = Vec::new(); + for q in [ + ListQuery::new().status("open").limit(5), + ListQuery::new() + .status("open") + .filter(NextFilter::new().with_label("waiting")) + .limit(2), + ] { + let reads = read::list_reads(&cfg, &q); + assert!(reads.select.is_some() && reads.ranking.is_none() && !reads.next); + let before = t.requests(); + let server = ready(records::list_board(&t, &reads)).unwrap(); + let after = t.requests(); + let page = read::list_board_page(&cfg, &server, &q).unwrap(); + let want = read::list_page(&cfg, &s, &q).unwrap(); + assert_eq!(ids(&page.items), ids(&want.items)); + assert_eq!(page.total, want.total, "the total counts every kept item"); + assert_eq!( + server.items.len(), + page.items.len(), + "only the page is held" + ); + costs.push(( + after.gets - before.gets, + after.batch_gets - before.batch_gets, + after.queries - before.queries, + after.index_queries - before.index_queries, + )); + } + costs + }; + let small = cost(0); + // The watermark; one `items#` Query; per page row `.adj#`, `.ladj#`, `custom_fields#`; its + // timestamps in one batch. With a label filter, each open row's labels first (12), then the + // page's adjacency and custom values. + assert_eq!(small, [(1, 1, 1 + 3 * 5, 0), (1, 1, 1 + 12 + 2 * 2, 0)]); + assert_eq!(cost(300), small, "300 closed items cost nothing more"); +} + +#[test] +fn a_ranked_list_with_next_and_a_graph_costs_what_its_scope_holds() { + // Review of PR #55, Test #7: a ranked list of persons whose keys read the `next` rank of + // projects and the `list` rank of goals reads next's board, the goals' scope and the graph. + // Its cost follows what those hold — items of no listed type, not active, on no edge, cost + // nothing. + let folder = Folder::platform().unwrap(); + let cfg = cfg(); + let q = ListQuery::new() + .sort(SortKey::Rank) + .now(NOW) + .filter(NextFilter::new().with_type("person")); + let reads = read::list_reads(&cfg, &q); + assert!(reads.next, "{reads:?}"); + let graph_reads = reads.ranking.clone().expect("a graph"); + assert!(!graph_reads.list_types.is_empty(), "{reads:?}"); + let cost = |unrelated: usize| { + let mut s = Store::open_in_memory(1); + for i in 0..4 { + s.create_item(&format!("ab12.u{i:03}"), "person", "P", "u"); + s.create_item(&format!("ab12.a{i:03}"), "action", "A", "u"); + s.add_edge( + &format!("ab12.a{i:03}"), + &format!("ab12.u{i:03}"), + EdgeKind::Mentions, + "u", + ); + } + s.create_item("ab12.g000", "goal", "G", "u"); + s.create_item("ab12.p000", "project", "P", "u"); + s.add_edge("ab12.p000", "ab12.g000", EdgeKind::ContributesTo, "u"); + s.add_edge("ab12.u000", "ab12.g000", EdgeKind::ContributesTo, "u"); + s.add_edge("ab12.u001", "ab12.p000", EdgeKind::ContributesTo, "u"); + s.add_label("ab12.a000", "waiting", "u"); + for i in 0..unrelated { + let id = format!("ab12.x{i:03}"); + s.create_item(&id, "action", "Done", "u"); + s.set_field(&id, "status", Some("closed".into()), "u"); + } + let t = MemTable::new("stream-1"); + ready(folder.fold_batch(&t, &delivered(&s, 3))).unwrap(); + let before = t.requests(); + let server = ready(records::list_board(&t, &reads)).unwrap(); + let after = t.requests(); + let page = read::list_board_page(&cfg, &server, &q).unwrap(); + assert_eq!( + ids(&page.items), + ids(&read::list_page(&cfg, &s, &q).unwrap().items) + ); + let graph = server.ranking.as_ref().unwrap(); + assert!(!graph.edges.is_empty() && !graph.listed.is_empty()); + let spent = ( + after.gets - before.gets, + after.batch_gets - before.batch_gets, + after.queries - before.queries, + after.index_queries - before.index_queries, + ); + // Every point read is the watermark or a batch; one index Query per listed type, the + // `active` index and one per graph list type. + assert_eq!(spent.0, 1); + assert!(spent.1 <= 3, "{spent:?}"); + assert_eq!(spent.3, 1 + 1 + graph_reads.list_types.len() as u64); + spent + }; + let small = cost(0); + assert_eq!( + cost(200), + small, + "200 unrelated closed actions cost nothing more" + ); +} + +#[test] +fn an_item_whose_typed_attribute_lags_its_row_is_not_listed_under_its_old_type() { + // Review of PR #55, Code #2 / Test #3: the fold writes `nxf_typed` after the row, so a run that + // died between the two leaves a row retyped or deleted whose attribute still names its old + // type. The list asks the index's copy of the row, not the attribute alone. + use nxs_fold_ddb::table::{text, Table}; + use nxs_fold_ddb::write::{Cond, Write}; + use nxs_foundation::change::Cell; + let folder = Folder::platform().unwrap(); + let cfg = cfg(); + let mut s = Store::open_in_memory(1); + for i in 0..3 { + s.create_item(&format!("ab12.u{i:03}"), "person", "P", "u"); + } + let t = MemTable::new("stream-1"); + ready(folder.fold_batch(&t, &delivered(&s, 1))).unwrap(); + for (id, column, value) in [("ab12.u001", "type", "goal"), ("ab12.u002", "deleted", "1")] { + ready(t.write(&Write { + sk: format!("items#{id}"), + set: vec![(column.into(), Cell::Text(value.into()))], + remove: Vec::new(), + condition: Some(Cond::RowPresent), + })) + .unwrap(); + } + let on_index = |ty: &str| -> Vec { + ready(t.query_typed(ty)) + .unwrap() + .iter() + .map(|r| text(r, "id").unwrap().to_string()) + .collect() + }; + assert_eq!( + on_index("person").len(), + 3, + "the attributes still name `person`" + ); + let listed = |ty: &str| -> Vec { + let q = ListQuery::new() + .sort(SortKey::Rank) + .filter(NextFilter::new().with_type(ty)); + let board = ready(records::list_board(&t, &read::list_reads(&cfg, &q))).unwrap(); + board.items.into_keys().collect() + }; + assert_eq!(listed("person"), ["ab12.u000"]); + assert!( + listed("goal").is_empty(), + "nor under its new type, until the attribute moves" + ); +} diff --git a/crates/fold-ddb/tests/records_differential.rs b/crates/fold-ddb/tests/records_differential.rs index 99ca095..b300b93 100644 --- a/crates/fold-ddb/tests/records_differential.rs +++ b/crates/fold-ddb/tests/records_differential.rs @@ -22,7 +22,7 @@ use nxs_fold_ddb::{board, records}; use nxs_foundation::change::Cell; mod common; -use common::{delivered, id, merged, serve, Lcg, NOW}; +use common::{delivered, head, id, merged, serve, Lcg, NOW}; /// A plugin that declares the generator's custom field, so the `custom` join is exercised, and /// ranks by priority and due date. @@ -440,7 +440,7 @@ fn a_stream_folded_before_the_entries_is_refused_then_refolded_and_reads_its_lab // stream and fold it again from position 0. No step of its own. let mark = ready(Folder::watermark(&t)).unwrap().unwrap(); assert!(!folder.is_current(&mark), "seed {seed}"); - ready(Folder::clear(&t)).unwrap(); + ready(Folder::clear(&t, head(&ops))).unwrap(); ready(folder.fold_batch(&t, &ops)).unwrap(); let mark = ready(Folder::watermark(&t)).unwrap().unwrap(); assert!(folder.is_current(&mark), "seed {seed}"); @@ -634,7 +634,7 @@ fn a_stream_folded_by_index_revision_1_is_refused_then_refolded_and_reads_its_co } let mark = ready(Folder::watermark(&t)).unwrap().unwrap(); assert!(!folder.is_current(&mark), "seed {seed}"); - ready(Folder::clear(&t)).unwrap(); + ready(Folder::clear(&t, head(&ops))).unwrap(); ready(folder.fold_batch(&t, &ops)).unwrap(); let served = served_board(&t); diff --git a/crates/fold-ddb/tests/robustness.rs b/crates/fold-ddb/tests/robustness.rs index 815e3b1..46279a3 100644 --- a/crates/fold-ddb/tests/robustness.rs +++ b/crates/fold-ddb/tests/robustness.rs @@ -1,22 +1,28 @@ //! What a server fold does when things go wrong (review of PR #24): a run that dies at any write //! and folds its batch again, an op the table cannot hold, a clear that dies half way, a release //! with other fold revisions, a lane cut by a limit, a snapshot that does not fit — and the `typed` -//! index kept through retypes, archives and deletes (6j6v.xkr7). +//! index kept through retypes, archives and deletes (6j6v.xkr7); a stream being refolded, a row +//! with no room for its index attributes and the escaping of a `typed` partition (review of PR +//! #55). mod common; -use common::{delivered, merged, NOW}; +use common::{delivered, head, merged, NOW}; use nexus_flow_core::model::EdgeKind; use nexus_flow_core::store::Store; +use nexus_flow_facade::read; use nxs_fold_ddb::board; use nxs_fold_ddb::fold::{Folded, Folder}; -use nxs_fold_ddb::layout::WATERMARK_KEY; +use nxs_fold_ddb::layout::{typed_partition, MAX_ITEM_BYTES, MAX_PARTITION_BYTES, WATERMARK_KEY}; use nxs_fold_ddb::mem::{ready, MemTable}; +use nxs_fold_ddb::records::{self, RecordsError}; use nxs_fold_ddb::snapshot::{self, ImportError}; use nxs_fold_ddb::table::{text, Dated, Outcome, Row, Table}; +use nxs_fold_ddb::write::attribute_bytes; use nxs_fold_ddb::write::{Cond, Write}; use nxs_foundation::change::Cell; use nxs_foundation::model::Op; +use std::collections::BTreeMap; use std::sync::atomic::{AtomicUsize, Ordering}; #[derive(Debug)] @@ -73,6 +79,10 @@ impl Table for Failing<'_> { Ok(ok(self.inner.get(sk).await)) } + async fn get_many(&self, sks: &[String]) -> Result, Injected> { + Ok(ok(self.inner.get_many(sks).await)) + } + async fn write(&self, write: &Write) -> Result { self.tick()?; Ok(ok(self.inner.write(write).await)) @@ -196,7 +206,8 @@ fn an_op_the_table_cannot_hold_is_refused_and_the_stream_folds_on() { let refused = ready(folder.fold_batch(&t, &ops)).unwrap(); let reasons: Vec<&str> = refused.iter().map(|r| r.reason.as_str()).collect(); // The long id's own three field ops and its edge (whose adjacency key is too long); the 500 KB - // description; and the design that would grow ab12.0002 past the limit. + // description; and the design that would grow ab12.0002 past the limit — weighed with the row + // it lands on before anything is written (review of PR #55, Integrity #4). assert_eq!(refused.len(), 6, "{reasons:#?}"); assert_eq!( reasons.iter().filter(|r| r.contains("a key of")).count(), @@ -204,7 +215,12 @@ fn an_op_the_table_cannot_hold_is_refused_and_the_stream_folds_on() { "{reasons:#?}" ); assert!(reasons.iter().any(|r| r.contains("a row of"))); - assert!(reasons.iter().any(|r| r.contains("kept their state"))); + assert!( + reasons + .iter() + .any(|r| r.contains("item ab12.0002 would weigh")), + "{reasons:#?}" + ); // The stream folded on: the watermark moved past every op, and what fits is there. let mark = ready(Folder::watermark(&t)).unwrap().unwrap(); assert_eq!(mark.folded_through, ops.len() as i64); @@ -312,6 +328,29 @@ fn retyping_archiving_and_deleting_move_an_item_on_the_typed_index() { step(&s), [none.clone(), v(&["ab12.0001"]), none.clone(), none.clone()] ); + // Created again under a deleted id, and a type cleared to none: the index lists what the store + // holds live, by type — whatever the store makes of the re-create — and an item without a type + // is on no partition (review of PR #55, Test #6). + s.create_item("ab12.0002", "person", "Bob again", "u"); + s.set_field("ab12.0001", "type", None, "u"); + let live = |s: &Store| { + ["person", "project", "a#b", "a"].map(|ty| { + let mut ids: Vec = s + .list_items() + .unwrap() + .into_iter() + .filter(|i| i.deleted.as_deref() != Some("1")) + .filter(|i| i.item_type.as_deref() == Some(ty)) + .map(|i| i.id) + .collect(); + ids.sort(); + ids + }) + }; + let want = live(&s); + assert_eq!(want[1], none, "the type cleared to none leaves `project`"); + assert_eq!(step(&s), want); + assert!(!t.rows()["items#ab12.0001"].contains_key("nxf_typed")); // The attribute is on the item rows only, never on an adjacency entry or another row. assert!(t .rows() @@ -341,16 +380,27 @@ fn a_clear_that_dies_keeps_the_watermark_and_a_release_with_other_rules_refolds_ let mark = ready(Folder::watermark(&t)).unwrap().unwrap(); assert!(!folder.is_current(&mark)); - // A clear that dies half way leaves the watermark — the next run sees the old revisions again. - assert!(ready(Folder::clear(&Failing::new(&t, 3))).is_err()); + // A clear that dies half way leaves a watermark that is not current — the next run clears it + // again. + assert!(ready(Folder::clear(&Failing::new(&t, 3), head(&ops))).is_err()); let after = ready(Folder::watermark(&t)).unwrap().unwrap(); assert!(!folder.is_current(&after)); - // Cleared for good and folded from position 0, it holds what a fresh fold holds. - ready(Folder::clear(&t)).unwrap(); - assert!(t.rows().is_empty()); + // Cleared for good, only the sentinel watermark is left; folded from position 0, the stream + // holds what a fresh fold holds, its watermark current and caught up with the clear's head. + ready(Folder::clear(&t, head(&ops))).unwrap(); + assert_eq!(t.rows().keys().collect::>(), [WATERMARK_KEY]); ready(folder.fold_batch(&t, &ops)).unwrap(); - assert_eq!(t.rows(), clean.rows()); + let without_mark = |t: &MemTable| { + let mut rows = t.rows(); + rows.remove(WATERMARK_KEY); + rows + }; + assert_eq!(without_mark(&t), without_mark(&clean)); + let mark = ready(Folder::watermark(&t)).unwrap().unwrap(); + assert!(folder.is_current(&mark)); + assert_eq!(mark.folded_through, head(&ops)); + assert_eq!(mark.refold_until, Some(head(&ops))); } #[test] @@ -488,11 +538,17 @@ fn a_snapshot_that_does_not_fit_is_refused_before_anything_is_written() { column.tables.get_mut("items").unwrap()[0].insert("nope".into(), serde_json::json!("x")); assert!(matches!(refused(&column), ImportError::Layout(_))); - // An import that died after some rows — no watermark yet — is simply run again. + // An import that died after some rows is refused to readers — its watermark is the sentinel + // it wrote first, naming no revision — and is simply run again. let t = MemTable::new("stream-1"); let partial = ready(snapshot::import(&Failing::new(&t, 10), &folder, &good)); assert!(partial.is_err()); - assert!(ready(Folder::watermark(&t)).unwrap().is_none()); + let mark = ready(Folder::watermark(&t)).unwrap().unwrap(); + assert!(mark.revisions.is_empty() && !folder.is_current(&mark)); + assert!(matches!( + ready(records::list_board(&t, &read::ListReads::default())), + Err(RecordsError::NotCurrent { found: None }) + )); ready(snapshot::import(&t, &folder, &good)).unwrap(); let whole = MemTable::new("stream-1"); ready(snapshot::import(&whole, &folder, &good)).unwrap(); @@ -502,3 +558,176 @@ fn a_snapshot_that_does_not_fit_is_refused_before_anything_is_written() { Folded::Folded )); } + +/// What the three readers of a hosted board answer for `t`: the records, the list, the ranking +/// graph (`plain` read before). +fn served( + t: &MemTable, + plain: &read::ActiveBoard, +) -> [Result<(), RecordsError>; 3] { + let selection = ready(board::select(t)).unwrap(); + [ + ready(records::active_board(t, selection)).map(drop), + ready(records::list_board(t, &read::ListReads::default())).map(drop), + ready(records::with_ranking( + t, + plain.clone(), + &read::RankingReads::default(), + )) + .map(drop), + ] +} + +#[test] +fn a_refolding_stream_is_refused_from_its_clear_until_the_refold_reaches_the_head() { + // Review of PR #55, Integrity #2: between a clear and the end of its refold the tables hold part + // of the history. Every reader refuses the stream for that whole window — `NotCurrent` while + // the watermark names no revision, `Refolding` once the first batch wrote them — and serves it + // again exactly when the refold reaches the head the clear named. + let folder = Folder::platform().unwrap(); + let s = merged(4); + let ops = delivered(&s, 4); + let t = MemTable::new("stream-1"); + ready(folder.fold_batch(&t, &ops)).unwrap(); + let plain = ready(records::active_board(&t, ready(board::select(&t)).unwrap())).unwrap(); + assert!(!plain.tickets.is_empty()); + assert!(served(&t, &plain).iter().all(Result::is_ok)); + let whole = t.rows(); + + // A clear that dies after its sentinel: refused, and not current, so the next run clears again. + assert!(ready(Folder::clear(&Failing::new(&t, 1), head(&ops))).is_err()); + for answer in served(&t, &plain) { + assert!( + matches!(answer, Err(RecordsError::NotCurrent { found: None })), + "{answer:?}" + ); + } + assert!(!folder.is_current(&ready(Folder::watermark(&t)).unwrap().unwrap())); + + ready(Folder::clear(&t, head(&ops))).unwrap(); + for answer in served(&t, &plain) { + assert!( + matches!(answer, Err(RecordsError::NotCurrent { found: None })), + "{answer:?}" + ); + } + + // Half the history refolded: this build's revisions are written, the head is not reached. + let half = ops.len() / 2; + ready(folder.fold_batch(&t, &ops[..half])).unwrap(); + assert!(folder.is_current(&ready(Folder::watermark(&t)).unwrap().unwrap())); + for answer in served(&t, &plain) { + match answer { + Err(RecordsError::Refolding { + folded_through, + until, + }) => assert_eq!((folded_through, until), (head(&ops[..half]), head(&ops))), + other => panic!("a half-refolded stream must be refused, got {other:?}"), + } + } + + // Caught up: served again, the same tables as before the clear; a batch folded twice changes + // nothing. + ready(folder.fold_batch(&t, &ops[half..])).unwrap(); + assert!(served(&t, &plain).iter().all(Result::is_ok)); + ready(folder.fold_batch(&t, &ops[half..])).unwrap(); + assert!(served(&t, &plain).iter().all(Result::is_ok)); + let without_mark = |mut rows: BTreeMap| { + rows.remove(WATERMARK_KEY); + rows + }; + assert_eq!(without_mark(t.rows()), without_mark(whole)); +} + +#[test] +fn a_typed_partition_escapes_the_type_so_no_stream_and_type_share_one() { + // Review of PR #55, Test #2: the stream is the partition's prefix and the type its rest, so a + // `#` in the type must not read as the separator. + assert_ne!(typed_partition("a", "b#c"), typed_partition("a#b", "c")); + assert_eq!(typed_partition("a", "b#c"), "a#b%23c"); + assert_ne!(typed_partition("a", "b%23c"), typed_partition("a", "b#c")); + + // A type of `#`s triples when escaped: it is weighed escaped, so a type whose own bytes fit is + // refused once its partition would not. + let folder = Folder::platform().unwrap(); + let hashes = "#".repeat(700); + assert!("stream-1#".len() + hashes.len() <= MAX_PARTITION_BYTES); + assert!(typed_partition("stream-1", &hashes).len() > MAX_PARTITION_BYTES); + let mut s = Store::open_in_memory(1); + s.create_item("ab12.0001", &hashes, "Hashes", "u"); + s.create_item("ab12.0002", &"x".repeat(700), "Plain", "u"); + let t = MemTable::new("stream-1"); + let refused = ready(folder.fold_batch(&t, &numbered(&s, 0))).unwrap(); + let reasons: Vec<&str> = refused.iter().map(|r| r.reason.as_str()).collect(); + assert_eq!(refused.len(), 1, "{reasons:#?}"); + assert!( + reasons[0].contains("`typed` index partition"), + "{reasons:#?}" + ); + assert!(ready(t.query_typed(&hashes)).unwrap().is_empty()); + let plain: Vec = ready(t.query_typed(&"x".repeat(700))) + .unwrap() + .iter() + .map(|r| text(r, "id").unwrap().to_string()) + .collect(); + assert_eq!(plain, ["ab12.0002"]); +} + +#[test] +fn an_item_with_no_room_for_its_index_attributes_is_refused_rather_than_left_off_an_index() { + // Review of PR #55, Integrity #4: closing a ticket swaps its `active` attribute for the two + // `dated` ones, which weigh more. A row the close itself fits, but not with those, used to take + // the close and drop the attribute write in silence — closed in its cells, still on `active`, + // on no `dated` lane. The op is refused before it writes anything instead. + let folder = Folder::platform().unwrap(); + let stream = "stream-1"; + let ops_with = |description: usize| { + let mut s = Store::open_in_memory(1); + s.create_item("ab12.0001", "task", "T", "u"); + s.set_field( + "ab12.0001", + "description", + Some("d".repeat(description)), + "u", + ); + s.set_field("ab12.0001", "status", Some("closed".into()), "u"); + numbered(&s, 0) + }; + let weigh = |r: &Row| r.iter().map(|(n, c)| attribute_bytes(n, c)).sum::(); + + // Measured on a small row: the closed row with its index attributes (`typed`, `dated`), and as + // it would weigh still carrying the open row's (`typed`, `active`). + let small = MemTable::new(stream); + ready(folder.fold_batch(&small, &ops_with(10))).unwrap(); + let row = small.rows()["items#ab12.0001"].clone(); + let dated: usize = ["nxf_dated", "nxf_dated_at"] + .iter() + .map(|a| attribute_bytes(a, &row[*a])) + .sum(); + let active = attribute_bytes("nxf_active", &Cell::Text(stream.into())); + assert!(dated > active, "the close makes the row heavier"); + let with_open_attributes = weigh(&row) - dated + active; + + // The description that leaves the closed row exactly at the limit with the open row's + // attributes: the close itself fits, its index attributes do not. + let ops = ops_with(10 + MAX_ITEM_BYTES - with_open_attributes); + let t = MemTable::new(stream); + let (close, before) = ops.split_last().unwrap(); + assert!(ready(folder.fold_batch(&t, before)).unwrap().is_empty()); + match ready(folder.fold(&t, &close.1)).unwrap() { + Folded::Refused(r) => assert!(r.reason.contains("index attributes"), "{r:?}"), + other => panic!("the close must be refused, got {other:?}"), + } + // Refused whole: still open in its cells and on its indexes, on no `dated` lane. + let row = &t.rows()["items#ab12.0001"]; + assert_eq!(text(row, "status"), Some("open")); + assert!(row.contains_key("nxf_active") && row.contains_key("nxf_typed")); + assert!(weigh(row) <= MAX_ITEM_BYTES); + assert!(ready(board::dated(&t, Dated::Closed, None)) + .unwrap() + .is_empty()); + assert_eq!( + ready(board::select(&t)).unwrap().board.lanes(NOW).ready, + vec!["ab12.0001"] + ); +} diff --git a/docs/specs/E4-sync-protocol-server.md b/docs/specs/E4-sync-protocol-server.md index 9dc3d5c..01d63b0 100644 --- a/docs/specs/E4-sync-protocol-server.md +++ b/docs/specs/E4-sync-protocol-server.md @@ -304,14 +304,43 @@ relay — does it with `nxs-fold-ddb`: the reducers a local replica registers, t reducers' rows — flags, adjacency entries, entries under a ticket — is versioned in the watermark as `revision_index` (1: the entries under a ticket). A stream folded before has no such revision, so `Folder::is_current` is false and the fold run clears and refolds it from position 0, exactly - as for a release with other reducer revisions — no manual step. Until then `active_board` - refuses the stream (`RecordsError::NotCurrent`) rather than answer without its labels. + as for a release with other reducer revisions — no manual step. Until then every records read + (`active_board`, `with_ranking`, `list_board`) refuses the stream (`RecordsError::NotCurrent`) + rather than answer without its labels — and goes on refusing it through the refold (review of PR + #55): `Folder::clear(table, until)` rewrites the watermark FIRST as a sentinel that names no + revision, `folded_through` 0 and `refold_until` = `until`, the relay head the refold must reach; + the refold's first batch writes the revisions, and the reads answer `RecordsError::Refolding` + until `folded_through` reaches `until`. Nothing has to be written for that, so a refold that + dies and resumes, or folds a batch twice, ends served the same. A snapshot import writes the same + sentinel before its first row. **No mixed fold versions:** `is_current` is equality, so a build + with older rules also clears a stream folded by newer ones — roll a revision bump out to every + fold run at once, and roll back the same way. +- **What a hosted read costs, per call** (review of PR #55, Integrity #1). One `GetItem` of the + watermark per read. `active_board`: `4n + p` more for `n` active tickets and `p` parents + outside them. `with_ranking`: one Query per active ticket, the neighbours' rows by `BatchGetItem` + (100 keys a request, unprocessed keys sent again), and per live neighbour up to three Queries; + a `next` key that reads a `[ranking.list]` rank adds one `typed` Query per such type and up to + three Queries per live, unarchived item of it. `list_board`: one `typed` Query per listed type + (or one `items#` Query, paged at 1 MB, without a type filter); a list that is NOT ranked keeps + and cuts on the rows first (`ListReads::select`), so beyond the rows it costs three Queries per + page row, one per kept row for a container or label filter, and one `BatchGetItem` per 100 + timestamps and outside parents — `list --limit 10` about 33 requests, whatever the stream + holds. A RANKED list reads three Queries per live, unarchived item of its types plus the batched + point reads, and `next`'s board and the graph when its keys read them: a rank is computed over + the type's whole `list` scope (`docs/specs/derived-ranking-keys.md` §12), so that cost is the + scope's, not the answer's. No read caps its requests; a hard per-call budget that answers with an + explicit error is filed as `6j6v.wq9e`. At most 16 items are read at a time + (`READ_CONCURRENCY`). - **Index reads are eventually consistent**; row reads are strong. The lanes right after a fold may be those of the state before it. - **An op the table cannot hold** (a sort key past 1,024 bytes, a row past 400 KB) is refused and reported, not retried for ever; the relay keeps it. A row that only outgrows the limit through several ops keeps its state at the write that would break it — the one result that depends on - arrival order, and only for rows a local replica can hold but DynamoDB cannot. + arrival order, and only for rows a local replica can hold but DynamoDB cannot. An item row is + weighed BEFORE the op writes anything — stored row, the op's writes and the index attributes it + would then carry (`nxf_active`, `nxf_typed`, `nxf_dated`/`nxf_dated_at`), one `GetItem` — so an + item never takes a change its index attributes do not fit beside and drops off an index in + silence (review of PR #55, Integrity #4). - **Edge flags are rewritten before the ticket's own flag**, so a run that dies between the two still sees the flip when it folds the op again. - **No Scan.** Every read names its partition. The `dynamodb-local` CI job records every request a @@ -323,7 +352,9 @@ relay — does it with `nxs-fold-ddb`: the reducers a local replica registers, t **The watermark** (`.meta#watermark`) is `folded_through` — a relay position, per decision 4 above — plus each reducer's fold revision. A batch folds, then moves the mark forward; a run that dies in between folds part of a batch again, which changes nothing. A release whose revisions differ clears -the stream — the watermark last, so a clear that dies is done again — and folds it from position 0. +the stream — the watermark first, rewritten as a sentinel without revisions, so a clear that dies +is done again and no reader serves the stream until its refold reaches the head the clear named — +and folds it from position 0. **A snapshot is tables plus watermark, without the log** — only to start a fold run. It is taken without a lock, the watermark first and the tables after, so the tables may hold ops past the mark diff --git a/docs/specs/derived-ranking-keys.md b/docs/specs/derived-ranking-keys.md index 3682d4f..536cdc2 100644 --- a/docs/specs/derived-ranking-keys.md +++ b/docs/specs/derived-ranking-keys.md @@ -159,12 +159,17 @@ derived key pays nothing. removed, kept up to date when the remove is folded. So one Query on an item's prefix reads every edge touching it, in and out, with its presence: inbound `contributes_to`, `mentions` both ways, and the `parent` edges a child's current parent is picked from. No new index, no new key - attribute: the TABLE CONTRACT IS UNCHANGED. Revision 3 makes every stream folded before it not - current; it answers `NotCurrent` until the fold run has folded it again. -- **What the server reads** (`records::with_ranking`): one Query per active ticket on its - adjacency; per neighbour that is not active one `GetItem` of its row and, when it is live, one - Query on its own adjacency for its current parent, plus one Query each for its labels and custom - values when a key reads them. + attribute: 1c5s alone leaves the table contract unchanged — the release that ships it changes it + all the same, with xkr7's GSI `typed` below. Revision 3 makes every stream folded before it not + current; it answers `NotCurrent` from the moment the fold run clears it, and `Refolding` until + the refold reaches the relay position the clear named (review of PR #55: `Folder::clear` leaves + a sentinel watermark without revisions and that position, so a half-refolded stream is never + served). +- **What the server reads** (`records::with_ranking`): one `GetItem` of the watermark (the read + refuses a stream that is not current or still refolding itself); one Query per active ticket on + its adjacency; the rows of the neighbours that are not active by `BatchGetItem` (100 keys a + request) and, per live one, one Query on its own adjacency for its current parent, plus one Query + each for its labels and custom values when a key reads them. - **A `[ranking.list]` rank in a `next` key** (refused on the server in 1c5s, ranked since xkr7): a `[ranking.next]` order that reads — directly or down its type chain — the rank of a type only `[ranking.list]` orders (§12, first point). That rank is computed over the type's live, @@ -178,10 +183,10 @@ derived key pays nothing. follows or counts). What xkr7 adds is the `typed` index (the `list` scope of a type), the read that ranks a type's live items from it, and lifting the refusal above. -**`list` needs a second index (decided §10.4).** Objects that `list` ranks, such as a person, are +**`list` needs another index (decided §10.4).** Objects that `list` ranks, such as a person, are long-lived and have no "active" status. If they did, they would rank through `next`, and `list` would not be needed for them. So the server cannot rank them from the `active` index. The table -gains a second sparse index, keyed by type: +gains a third sparse index — after `active` and `dated` —, keyed by type: | what | name | type | role | |---|---|---|---| @@ -221,7 +226,18 @@ gains a second sparse index, keyed by type: by the code `list_page` runs, and `read::list_board` reads the same board from a store, the parity oracle. - **Without a type filter** the server lists every live item: it reads the partition's `items#` - rows (strongly consistent, one Query, no index), since the index lists one type at a time. + rows (strongly consistent, one Query — paged by DynamoDB at 1 MB —, no index), since the index + lists one type at a time. +- **What a list costs** (review of PR #55, Integrity #1). A `list` that is not ranked orders and + filters by the item rows alone, so `ListReads::select` lets the server keep and cut BEFORE the + joins: status and type are asked of the row, a container or label filter reads the one join it + needs per kept row, and only the page (at most `limit` rows) reads the rest — three Queries a + row, the timestamps and outside parents by `BatchGetItem`. `list --limit 10` costs the rows' + Query and about 33 requests, however many items the stream holds. A RANKED `list` reads every + live, unarchived item of its types in full — three Queries each — because a rank is computed + over the type's whole `list` scope (§12, "The scope of a `list` rank": every live, unarchived item of the type, + whatever the query's status, filter or limit). That cost is the scope's, not the answer's, and it + is not capped: a hard per-call budget is filed as `6j6v.wq9e` (E4 spec §5.1). - **Paging.** As for the hosted `next`: the store path's page cache is machine-local and a server's paging is an open decision (15ed), so the store-free `list` is not paged — `limit` cuts it with the total disclosed, `paginate` and a token are refused with `validation`, and a server @@ -310,7 +326,8 @@ the plugin declares (`plugin-custom-fields.md`), as decided in §10.2. 3. **Persons rank by `sum`** (§3.3), equal weights, `weight` left out until data asks for it. Strictly sequential keys were declined: one more mention would always beat any number of open waiting-fors. -4. **A hosted `list` with derived keys uses a second index** (§6, GSI `typed`). Ranking only active +4. **A hosted `list` with derived keys uses a second index** (§6, GSI `typed`; the table's third, + counting `dated`, which this decision did not count). Ranking only active items was declined: the objects `list` ranks have no active status by design. If they had one, they would rank through `next`, and `list` would not be needed. Leaving `list` unranked on the server was declined because the product reads through the hosted path.