From a442d541f41e80fccff303aa3ec677c3806de235 Mon Sep 17 00:00:00 2001 From: Carsten Koch Date: Sun, 11 Oct 2026 18:29:24 +0200 Subject: [PATCH 1/2] Derived ranking keys on the hosted server for next (nxf 6j6v.1c5s) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit The store-free `next` (read::next_active) now ranks a plugin with `related` / `count` / `sum` keys from a RankingGraph on ActiveBoard, by the same evaluation the store path runs. Owner decision 2026-10-11: a key reads every LIVE neighbour (closed and archived count, deleted do not) on both paths. - facade: ActiveBoard::ranking, RankingReads, RankingGraph, Neighbour, next_ranking_reads, with_ranking (the parity oracle); derived.rs split into plan / context fill (store or graph) / finish. A [ranking.next] order reading the rank of a type only [ranking.list] orders is refused on the store-free path until xkr7. - fold-ddb: .adj# entries copy from_id/to_id/kind/lamport/site and a removed flag, kept on edge removes; INDEX_REVISION 2 -> 3 (old streams NotCurrent until refolded); records::with_ranking reads the graph. Table contract unchanged (no new GSI, no new key attribute). - core: EdgeKind and graph::Edge derive Ord. - tests: derived_differential (SQL = Board = fold-ddb on generated boards, determinism, request-count guard, revision-2 refold), facade seam tests for live neighbours and the list-rank refusal. - spec §6/§10.5/§11/§12 amended. Co-Authored-By: Claude Opus 5.5 Claude-Session: https://claude.ai/code/session_016q5uMUkZJQW4fckJoA3FME --- Cargo.lock | 1 + changes/1c5s-server-derived-keys.md | 8 + crates/core/src/graph.rs | 3 +- crates/core/src/model.rs | 2 +- crates/facade/src/read.rs | 210 +++++++- crates/facade/src/read/derived.rs | 405 ++++++++++++--- crates/facade/tests/derived_ranking.rs | 207 +++++++- crates/fold-ddb/Cargo.toml | 3 + crates/fold-ddb/src/board.rs | 77 ++- crates/fold-ddb/src/fold.rs | 8 +- crates/fold-ddb/src/lib.rs | 9 +- crates/fold-ddb/src/records.rs | 176 ++++++- crates/fold-ddb/tests/derived_differential.rs | 483 ++++++++++++++++++ docs/specs/derived-ranking-keys.md | 80 ++- 14 files changed, 1529 insertions(+), 143 deletions(-) create mode 100644 changes/1c5s-server-derived-keys.md create mode 100644 crates/fold-ddb/tests/derived_differential.rs diff --git a/Cargo.lock b/Cargo.lock index b695c61..e10c7f3 100644 --- a/Cargo.lock +++ b/Cargo.lock @@ -2442,6 +2442,7 @@ version = "0.209.0" dependencies = [ "aws-sdk-dynamodb", "futures-util", + "inventory", "nexus-chat", "nexus-flow-core", "nexus-flow-facade", diff --git a/changes/1c5s-server-derived-keys.md b/changes/1c5s-server-derived-keys.md new file mode 100644 index 0000000..1d63ec8 --- /dev/null +++ b/changes/1c5s-server-derived-keys.md @@ -0,0 +1,8 @@ +--- +type: added +facade: changed +--- +[en] +A hosted board now ranks `next` by derived ranking keys (`related`, `count`, `sum`) exactly as a local replica does. The store-free `next` (`read::next_active`, `next_active_value`, `next_active_filtered`) ranks a plugin with such keys from a new `ActiveBoard::ranking` graph: the edges around the active tickets and the live neighbours they reach, closed and archived ones included, deleted ones not. A server fills it with `nxs_fold_ddb::records::with_ranking` for what `read::next_ranking_reads(cfg)` names; `read::with_ranking` reads the same graph from a store. Without the graph, `next_active` still refuses such a plugin. One case stays refused on the store-free path: a `[ranking.next]` order that reads the rank of a type only `[ranking.list]` orders — answer that `next` from a store. Everything is additive: `ActiveBoard` gains a field (it is `#[non_exhaustive]`), and `RankingReads`, `RankingGraph`, `Neighbour`, `next_ranking_reads` and `with_ranking` are new. For a server's operators: the fold's index revision goes from 2 to 3, so every hosted stream is not current and is folded again on its next run; until then its records read answers `NotCurrent`. The table contract is unchanged: no new index and no new key attribute, only more attributes on the existing `.adj#` adjacency entries. +[de] +Ein gehostetes Brett rankt `next` jetzt nach abgeleiteten Ranking-Schlüsseln (`related`, `count`, `sum`) genau so wie ein lokales Replikat. Das speicherfreie `next` (`read::next_active`, `next_active_value`, `next_active_filtered`) rankt ein Plugin mit solchen Schlüsseln aus einem neuen Graphen `ActiveBoard::ranking`: den Kanten um die aktiven Tickets und den lebenden Nachbarn, die sie erreichen, geschlossene und archivierte eingeschlossen, gelöschte nicht. Ein Server füllt ihn mit `nxs_fold_ddb::records::with_ranking` für das, was `read::next_ranking_reads(cfg)` nennt; `read::with_ranking` liest denselben Graphen aus einem Speicher. Ohne den Graphen lehnt `next_active` ein solches Plugin weiterhin ab. Ein Fall bleibt auf dem speicherfreien Weg abgelehnt: eine Reihenfolge in `[ranking.next]`, die den Rang eines Typs liest, den nur `[ranking.list]` ordnet — dieses `next` beantwortet ein Speicher. Alles ist additiv: `ActiveBoard` bekommt ein Feld (es ist `#[non_exhaustive]`), und `RankingReads`, `RankingGraph`, `Neighbour`, `next_ranking_reads` und `with_ranking` sind neu. Für den Betrieb eines Servers: Die Index-Revision des Folds steigt von 2 auf 3, daher ist jeder gehostete Stream nicht aktuell und wird bei seinem nächsten Lauf neu gefaltet; bis dahin antwortet sein Records-Lesen mit `NotCurrent`. Der Tabellenvertrag bleibt unverändert: kein neuer Index und kein neues Schlüsselattribut, nur mehr Attribute an den bestehenden `.adj#`-Adjazenzeinträgen. diff --git a/crates/core/src/graph.rs b/crates/core/src/graph.rs index dca4ce6..34b8f84 100644 --- a/crates/core/src/graph.rs +++ b/crates/core/src/graph.rs @@ -30,7 +30,8 @@ pub struct Ticket { } /// A present edge, `from` → `to` (a `dep` points at its blocker, a `parent` edge at the parent). -#[derive(Debug, Clone, PartialEq, Eq)] +/// Ordered by `from`, `to`, `kind`, so a set of edges has one order whatever order they came in. +#[derive(Debug, Clone, PartialEq, Eq, PartialOrd, Ord)] pub struct Edge { pub from: String, pub to: String, diff --git a/crates/core/src/model.rs b/crates/core/src/model.rs index e00fb57..5beca2d 100644 --- a/crates/core/src/model.rs +++ b/crates/core/src/model.rs @@ -48,7 +48,7 @@ impl Status { /// the OR-set replacement for the former single-parent `belongs_to` LWW register. The substrate is /// n:m-capable like every other edge; single-parent is enforced at the write seam during ②a and /// relaxed in ②c. The single current parent is projected by the `present_parent` view. -#[derive(Debug, Clone, Copy, PartialEq, Eq, Hash)] +#[derive(Debug, Clone, Copy, PartialEq, Eq, PartialOrd, Ord, Hash)] pub enum EdgeKind { Dep, ContributesTo, diff --git a/crates/facade/src/read.rs b/crates/facade/src/read.rs index 47d775b..7784967 100644 --- a/crates/facade/src/read.rs +++ b/crates/facade/src/read.rs @@ -24,7 +24,7 @@ use crate::plugin::{Dir, Nulls, PluginConfig, RankKey, RankSpec, Verb}; use crate::record::{item_value, items_value, raw_field}; use nexus_flow_core::derive; use nexus_flow_core::graph::Board; -use nexus_flow_core::model::{ItemRow, LinkWeight, ThreadLink}; +use nexus_flow_core::model::{EdgeKind, ItemRow, LinkWeight, ThreadLink}; use nexus_flow_core::store::Store; use serde_json::{json, Value}; use std::collections::{BTreeMap, BTreeSet, HashMap, HashSet}; @@ -2734,6 +2734,11 @@ pub struct ActiveBoard { pub board: Board, /// By id. pub tickets: BTreeMap, + /// The graph the derived ranking keys of `[ranking.next]` read (6j6v.1c5s): `None` until a + /// reader fills it — [`with_ranking`] from a store, `nxs_fold_ddb::records::with_ranking` from + /// the folded tables — for what [`next_ranking_reads`] names. A plugin without derived `next` + /// keys never needs it; [`next_active`] refuses one with such keys while it is missing. + pub ranking: Option, } impl ActiveBoard { @@ -2745,6 +2750,7 @@ impl ActiveBoard { .into_iter() .map(|t| (t.item.id.clone(), t)) .collect(), + ranking: None, } } @@ -2780,6 +2786,163 @@ impl ActiveBoard { } } +/// What the derived ranking keys of a `[ranking.next]` read beyond the active tickets' records +/// (6j6v.1c5s) — what a reader of an [`ActiveBoard`] puts in its [`RankingGraph`]. +/// [`next_ranking_reads`] gives it for a plugin. +/// +/// `#[non_exhaustive]`: start from `RankingReads::default()` and set the fields. +#[derive(Debug, Clone, Default, PartialEq, Eq)] +#[non_exhaustive] +pub struct RankingReads { + /// The edge kinds the keys follow or count; `mentions` too when a key counts `mentioned_by`. + pub kinds: BTreeSet, + /// A count filters on a label: the neighbours' labels. + pub labels: bool, + /// An `of` reads a custom field: the neighbours' custom values. + pub custom: bool, +} + +impl RankingReads { + /// Whether a graph read for `self` holds everything a graph read for `needed` holds. + pub fn covers(&self, needed: &RankingReads) -> bool { + needed.kinds.is_subset(&self.kinds) + && (self.labels || !needed.labels) + && (self.custom || !needed.custom) + } +} + +/// The graph around a board's active tickets that the derived ranking keys read (6j6v.1c5s, +/// docs/specs/derived-ranking-keys.md §6): the input [`next_active`] ranks a `[ranking.next]` with +/// `related` / `count` / `sum` keys from, so a server ranks exactly as [`next`] does over a store. +/// +/// - `edges`: every present edge of a kind in `reads.kinds` with at least one ACTIVE end (an +/// [`ActiveBoard::tickets`] entry) and both ends LIVE (not deleted, not missing). A `parent` edge +/// is the `present_parent` projection — a child's one current parent, the one `belongs_to` reads — +/// so a child re-parented concurrently is under one parent only. Distinct, ordered. +/// - `neighbours`: the other ends of `edges` that are not active tickets — closed and archived +/// items included, since a key reads every LIVE neighbour (owner, 2026-10-11) — each with its row +/// (`belongs_to` filled as the store fills it), its labels when `reads.labels` and its non-empty +/// custom values when `reads.custom`, else empty. +/// +/// `#[non_exhaustive]`: build it with [`RankingGraph::new`] and set the fields. +#[derive(Debug, Clone, Default, PartialEq, Eq)] +#[non_exhaustive] +pub struct RankingGraph { + /// What the graph was read for. + pub reads: RankingReads, + pub edges: BTreeSet, + /// By id. + pub neighbours: BTreeMap, +} + +impl RankingGraph { + /// An empty graph read for `reads`. + pub fn new(reads: RankingReads) -> RankingGraph { + RankingGraph { + reads, + ..RankingGraph::default() + } + } +} + +/// A live item next to an active ticket that is not itself active: what a derived ranking key reads +/// of it ([`RankingGraph::neighbours`]). +/// +/// `#[non_exhaustive]`: build it with [`Neighbour::new`] and set the fields. +#[derive(Debug, Clone, PartialEq, Eq)] +#[non_exhaustive] +pub struct Neighbour { + pub item: ItemRow, + /// Present labels, sorted and distinct, as `present_labels`. + pub labels: Vec, + /// Every non-empty custom value, as the store reads them in bulk. + pub custom: BTreeMap, +} + +impl Neighbour { + /// A neighbour with no labels or custom values read. + pub fn new(item: ItemRow) -> Neighbour { + Neighbour { + item, + labels: Vec::new(), + custom: BTreeMap::new(), + } + } +} + +/// What the store-free `next` of `cfg` reads beyond the active tickets' records — the +/// [`RankingReads`] a reader fills [`ActiveBoard::ranking`] for — or `None` when `[ranking.next]` +/// declares no derived ranking key (6j6v.1c5s). It does not depend on the board. +/// +/// `validation` for a `[ranking.next]` that reads, directly or down its type chain, the rank of a +/// type only `[ranking.list]` orders (docs/specs/derived-ranking-keys.md §6): that rank is computed +/// over the type's live items, and a board of active tickets does not hold them. The store path +/// ([`next`]) ranks such a plugin. +pub fn next_ranking_reads(cfg: &PluginConfig) -> Result> { + derived::reads_for_next(cfg) +} + +/// `board` with its [`RankingGraph`] for `reads` read from `store` (6j6v.1c5s) — the parity oracle +/// for `nxs_fold_ddb::records::with_ranking`, as [`active_board`] is for the records. One edge pass +/// ([`Store::present_edges_of_kinds`]), one bulk read of the neighbours' rows, and one of their +/// labels and of their custom values when `reads` asks for them. +pub fn with_ranking( + store: &Store, + mut board: ActiveBoard, + reads: &RankingReads, +) -> Result { + let kinds: Vec = reads.kinds.iter().copied().collect(); + let active = |id: &str| board.tickets.contains_key(id); + let touching: Vec<(String, String, EdgeKind)> = store + .present_edges_of_kinds(&kinds)? + .into_iter() + .filter(|(from, to, _)| active(from) || active(to)) + .collect(); + let others: BTreeSet<&str> = touching + .iter() + .flat_map(|(from, to, _)| [from.as_str(), to.as_str()]) + .filter(|id| !active(id)) + .collect(); + let others: Vec<&str> = others.into_iter().collect(); + let live: BTreeMap = store + .get_items(&others)? + .into_iter() + .filter(|i| i.deleted.as_deref() != Some("1")) + .map(|i| (i.id.clone(), i)) + .collect(); + let edges: BTreeSet = touching + .into_iter() + .filter(|(from, to, _)| { + (active(from) || live.contains_key(from)) && (active(to) || live.contains_key(to)) + }) + .map(|(from, to, kind)| nexus_flow_core::graph::Edge { from, to, kind }) + .collect(); + let ids: Vec<&str> = live.keys().map(String::as_str).collect(); + let mut labels = if reads.labels { + store.labels_of_bulk(&ids)? + } else { + BTreeMap::new() + }; + let mut custom = if reads.custom { + store.custom_fields_of_bulk(&ids)? + } else { + BTreeMap::new() + }; + let mut graph = RankingGraph::new(reads.clone()); + graph.edges = edges; + graph.neighbours = live + .into_iter() + .map(|(id, item)| { + let mut n = Neighbour::new(item); + n.labels = labels.remove(&id).unwrap_or_default(); + n.custom = custom.remove(&id).unwrap_or_default(); + (id, n) + }) + .collect(); + board.ranking = Some(graph); + Ok(board) +} + /// The store path's input as an [`ActiveBoard`]: the store's selection ([`graph::select`]) and, for /// every active ticket, its row and joins, each read in bulk. What [`next_active`] and the other /// store-free lanes answer from it is what [`next`] and its siblings answer from the store. @@ -2823,21 +2986,44 @@ pub fn active_board(store: &Store) -> Result { /// /// [`Engine::next`]: crate::engine::Engine::next /// -/// A `[ranking.next]` with derived ranking keys (6j6v.x0wf) is refused with `validation`: an -/// [`ActiveBoard`] holds neither the edges nor the scopes those keys read, and ranking without them -/// would order apart from the store path in silence. The server path is its own cut (6j6v.1c5s). +/// A `[ranking.next]` with derived ranking keys (6j6v.x0wf) is ranked from the board's +/// [`RankingGraph`] (6j6v.1c5s), by the same evaluation the store path runs. It is refused with +/// `validation` when the board carries no graph, or one read for less than +/// [`next_ranking_reads`] names — ranking without those edges would order apart from the store path +/// in silence — and when [`next_ranking_reads`] refuses the plugin. pub fn next_active(cfg: &PluginConfig, board: &ActiveBoard, now: &str) -> Result> { crate::validate::iso_date(now)?; - if cfg.ranking.has_derived(Verb::Next) { - return Err(NxfError::validation(format!( - "plugin '{}' ranks `next` by derived ranking keys, which the store-free lanes do not \ - compute yet — answer `next` from a store", - cfg.name - ))); - } + let graph = match next_ranking_reads(cfg)? { + None => None, + Some(needed) => Some( + board + .ranking + .as_ref() + .filter(|g| g.reads.covers(&needed)) + .ok_or_else(|| { + NxfError::validation(format!( + "plugin '{}' ranks `next` by derived ranking keys, and the board was read \ + without the graph they read — fill `ActiveBoard::ranking` for \ + `next_ranking_reads` (`read::with_ranking` from a store, \ + `records::with_ranking` from the folded tables)", + cfg.name + )) + })?, + ), + }; let candidates = board.board.lanes(now).candidates; let items = board.items(candidates.iter().map(|c| &c.id)); - Ok(next_lane(cfg, candidates, items, DEFAULT_SORT_NEXT, None)) + let derived = match graph { + Some(graph) => derived::compute_active(cfg, board, graph, &items)?, + None => None, + }; + Ok(next_lane( + cfg, + candidates, + items, + DEFAULT_SORT_NEXT, + derived.as_ref(), + )) } /// [`next_active`], narrowed by `filter` (6j6v.15ed): the rows [`next_filtered`] gives over a store diff --git a/crates/facade/src/read/derived.rs b/crates/facade/src/read/derived.rs index 86e3e3a..9906834 100644 --- a/crates/facade/src/read/derived.rs +++ b/crates/facade/src/read/derived.rs @@ -19,10 +19,17 @@ //! //! So a query costs O(E + N log N) in the relevant edges and the items in scope, and a plugin //! that declares no derived key pays nothing: [`compute`] returns `None` before any read. +//! +//! **Without a store** (6j6v.1c5s). The store-free `next` ([`super::next_active`]) ranks a board a +//! server read: [`compute_active`] takes steps 1 and 2 from the board's [`RankingGraph`] — the +//! edges, already projected and filtered to live ends, and the live neighbours' rows, labels and +//! custom values — and runs steps 3 and 4 on the same [`Context`], so the two paths cannot evaluate +//! a key apart. A neighbour counts when it is LIVE, closed and archived ones included, on both +//! paths (spec §6, owner 2026-10-11). What the graph must hold is [`reads_for_next`]. -use super::{lanes, mentioners_of, resolve_items}; +use super::{lanes, mentioners_of, resolve_items, ActiveBoard, RankingGraph, RankingReads}; use crate::error::{NxfError, Result}; -use crate::plugin::{Dir, Nulls, Pick, PluginConfig, RankKey, RankOf, Verb}; +use crate::plugin::{Dir, Nulls, Pick, PluginConfig, RankKey, RankOf, RankSpec, Verb}; use crate::record::{raw_field, RANKABLE_FIELDS}; use nexus_flow_core::model::{EdgeKind, ItemRow}; use nexus_flow_core::store::Store; @@ -131,20 +138,20 @@ pub(super) fn list_scope(store: &Store) -> Result> { .collect()) } -/// The values the derived keys of `verb`'s policy rank `ranked` by — `None`, before any read, -/// when the policy has no derived key. `validation` when a `list` key reads the rank of a type -/// `next` ranks and the caller passed neither `next`'s candidates nor a reference time. -pub(super) fn compute( - cfg: &PluginConfig, - store: &Store, - verb: Verb, - ranked: &[ItemRow], - scopes: Scopes<'_>, -) -> Result> { +/// The derived side of one query, decided before any read: the type chain the ranked rows' orders +/// read the rank of (leaves first), and what the derived keys in play read. +struct Plan<'c> { + spec: &'c RankSpec, + chain: Vec, + needs: Needs, +} + +/// The [`Plan`] for ranking `ranked` under `verb` — `None` when the policy has no derived key. +fn plan<'c>(cfg: &'c PluginConfig, verb: Verb, ranked: &[ItemRow]) -> Option> { let ranking = &cfg.ranking; let spec = ranking.spec(verb); if !spec.has_derived() { - return Ok(None); + return None; } // The chain, leaves first: the orders the ranked rows' orders read the rank of, transitively. @@ -158,10 +165,77 @@ pub(super) fn compute( } } + // Every derived key in play. + let mut keys: Vec<&RankKey> = Vec::new(); + for ty in &ranked_types { + keys.extend(spec.order_for(*ty).iter().filter(|k| k.is_derived())); + } + for (v, ty) in &chain { + keys.extend( + ranking.spec(*v).by_type[ty] + .order + .iter() + .filter(|k| k.is_derived()), + ); + } + let needs = Needs::of(&keys); + Some(Plan { spec, chain, needs }) +} + +impl Plan<'_> { + /// The rows each order of the chain ranks: its type's rows in the scope of its verb. + fn node_rows<'r>(&self, next: &'r [ItemRow], list: &'r [ItemRow]) -> Vec> { + self.chain + .iter() + .map(|(v, ty)| { + let scope = match v { + Verb::Next => next, + Verb::List => list, + }; + scope + .iter() + .filter(|i| i.item_type.as_deref() == Some(ty.as_str())) + .collect() + }) + .collect() + } + + /// Whether some order of the chain is a `[ranking.list]` order. + fn reads_list(&self) -> bool { + self.chain.iter().any(|(v, _)| *v == Verb::List) + } +} + +/// Every row a key is evaluated on: the ranked rows and the rows of the chain's scopes. +fn subjects<'a>( + ranked: &'a [ItemRow], + node_rows: &[Vec<&'a ItemRow>], +) -> HashMap<&'a str, &'a ItemRow> { + let mut subjects: HashMap<&str, &ItemRow> = HashMap::new(); + for row in ranked.iter().chain(node_rows.iter().flatten().copied()) { + subjects.insert(row.id.as_str(), row); + } + subjects +} + +/// The values the derived keys of `verb`'s policy rank `ranked` by — `None`, before any read, +/// when the policy has no derived key. `validation` when a `list` key reads the rank of a type +/// `next` ranks and the caller passed neither `next`'s candidates nor a reference time. +pub(super) fn compute( + cfg: &PluginConfig, + store: &Store, + verb: Verb, + ranked: &[ItemRow], + scopes: Scopes<'_>, +) -> Result> { + let Some(plan) = plan(cfg, verb, ranked) else { + return Ok(None); + }; + // The scopes the chain ranks over, read only when the caller does not hold them. let next_owned; let next_rows: &[ItemRow] = match scopes.next { - _ if !chain.iter().any(|(v, _)| *v == Verb::Next) => &[], + _ if !plan.chain.iter().any(|(v, _)| *v == Verb::Next) => &[], Some(rows) => rows, None => { let now = scopes.now.ok_or_else(|| { @@ -178,89 +252,115 @@ pub(super) fn compute( }; let list_owned; let list_rows: &[ItemRow] = match scopes.list { - _ if !chain.iter().any(|(v, _)| *v == Verb::List) => &[], + _ if !plan.reads_list() => &[], Some(rows) => rows, None => { list_owned = list_scope(store)?; &list_owned } }; - let node_rows: Vec> = chain - .iter() - .map(|(v, ty)| { - let scope = match v { - Verb::Next => next_rows, - Verb::List => list_rows, - }; - scope - .iter() - .filter(|i| i.item_type.as_deref() == Some(ty.as_str())) - .collect() - }) - .collect(); + let node_rows = plan.node_rows(next_rows, list_rows); + let mut ctx = Context::new(cfg, subjects(ranked, &node_rows)); + ctx.read_edges(store, &plan.needs)?; + Ok(Some(ctx.finish(&plan, verb, ranked, &node_rows))) +} - // Every row a key is evaluated on, and every derived key in play. - let mut subjects: HashMap<&str, &ItemRow> = HashMap::new(); - for row in ranked.iter().chain(node_rows.iter().flatten().copied()) { - subjects.insert(row.id.as_str(), row); +/// [`compute`] for the store-free `next` (6j6v.1c5s): the values the derived keys of +/// `[ranking.next]` rank `candidates` — every candidate of the board — by, read from the +/// [`RankingGraph`] a reader filled instead of from a store. The candidates are `next`'s scope for +/// "rank", as on the store path. `None` when the policy has no derived key. +/// +/// The same [`Context`] evaluates the keys as on the store path; only where its edges, rows, +/// labels and custom values come from differs ([`Context::read_graph`]). A chain that reaches a +/// `[ranking.list]` order is refused before this is called ([`reads_for_next`]); it answers +/// `validation` here too rather than rank without that scope. +pub(super) fn compute_active( + cfg: &PluginConfig, + board: &ActiveBoard, + graph: &RankingGraph, + candidates: &[ItemRow], +) -> Result> { + let Some(plan) = plan(cfg, Verb::Next, candidates) else { + return Ok(None); + }; + if plan.reads_list() { + return Err(list_scope_refusal(cfg)); } - let mut keys: Vec<&RankKey> = Vec::new(); - for ty in &ranked_types { - keys.extend(spec.order_for(*ty).iter().filter(|k| k.is_derived())); + let node_rows = plan.node_rows(candidates, &[]); + let mut ctx = Context::new(cfg, subjects(candidates, &node_rows)); + ctx.read_graph(board, graph, &plan.needs); + Ok(Some(ctx.finish(&plan, Verb::Next, candidates, &node_rows))) +} + +/// What the store-free `next` of `cfg` reads beyond the active tickets' records (6j6v.1c5s): +/// `None` when `[ranking.next]` declares no derived key. Gathered over every order of the section, +/// so it does not depend on which types a board holds. +/// +/// `validation` when an order of `[ranking.next]` reads — directly or down its type chain — the +/// rank of a type only `[ranking.list]` orders (spec §12, first point): that rank is computed over +/// the live items of the type, and a server reads only the active ones (owner, 2026-10-11). +pub(super) fn reads_for_next(cfg: &PluginConfig) -> Result> { + let ranking = &cfg.ranking; + let spec = ranking.spec(Verb::Next); + if !spec.has_derived() { + return Ok(None); } - for (v, ty) in &chain { - keys.extend( - ranking.spec(*v).by_type[ty] - .order - .iter() - .filter(|k| k.is_derived()), - ); + let mut chain: Vec = Vec::new(); + let mut seen: BTreeSet = BTreeSet::new(); + for ty in spec.by_type.keys() { + post_order(cfg, (Verb::Next, ty.clone()), &mut seen, &mut chain); } - + if chain.iter().any(|(v, _)| *v == Verb::List) { + return Err(list_scope_refusal(cfg)); + } + let keys: Vec<&RankKey> = spec + .order + .iter() + .chain(spec.by_type.values().flat_map(|o| o.order.iter())) + .filter(|k| k.is_derived()) + .collect(); let needs = Needs::of(&keys); - let mut ctx = Context { - cfg, - subjects, - out: Adjacency::new(), - inb: Adjacency::new(), - mentioners: BTreeMap::new(), - neighbours: HashMap::new(), - labels: BTreeMap::new(), - custom: BTreeMap::new(), - ranks: HashMap::new(), - }; - ctx.read_edges(store, &needs)?; - - // Bottom-up: each order of the chain ranks its scope once; its dependents look the rank up. - for (node, rows) in chain.iter().zip(&node_rows) { - let keys = &ranking.spec(node.0).by_type[&node.1].order; - let values: Vec> = rows.iter().map(|r| ctx.values(keys, node.0, r)).collect(); - let mut order: Vec = (0..rows.len()).collect(); - let cmp = |a: usize, b: usize| { - super::keys_cmp_values(rows[a], rows[b], keys, Some(&values[a]), Some(&values[b])) - }; - order.sort_by(|a, b| cmp(*a, *b).then_with(|| rows[*a].id.cmp(&rows[*b].id))); - // A DENSE rank over the declared keys (spec §4): rows equal on every key share a rank, and - // the next distinct row takes the next integer. The id orders the sort, never the rank. - let mut ranks: HashMap = HashMap::with_capacity(rows.len()); - let mut rank = 0_i64; - for (n, &i) in order.iter().enumerate() { - if n > 0 && cmp(order[n - 1], i) != Ordering::Equal { - rank += 1; - } - ranks.insert(rows[i].id.clone(), rank); - } - ctx.ranks.insert(node.clone(), ranks); + let mut reads = RankingReads::default(); + reads.kinds.extend(needs.kinds.iter().copied()); + if needs.mentioned_by { + reads.kinds.insert(EdgeKind::Mentions); } + reads.labels = needs.labels; + reads.custom = needs.custom; + Ok(Some(reads)) +} - let values = ranked - .iter() - .map(|r| { - let keys = spec.order_for(r.item_type.as_deref()); - (r.id.clone(), ctx.values(keys, verb, r)) +/// The refusal of a `[ranking.next]` whose type chain reaches a `[ranking.list]` order, on the +/// store-free path. +fn list_scope_refusal(cfg: &PluginConfig) -> NxfError { + let ranking = &cfg.ranking; + let reached: Vec = ranking + .spec(Verb::Next) + .by_type + .keys() + .flat_map(|ty| { + let mut chain = Vec::new(); + post_order( + cfg, + (Verb::Next, ty.clone()), + &mut BTreeSet::new(), + &mut chain, + ); + chain + .into_iter() + .filter(|(v, _)| *v == Verb::List) + .map(move |(_, list_ty)| { + format!("[ranking.next.{ty}] reads the rank of '{list_ty}'") + }) }) .collect(); - Ok(Some(Derived { values })) + NxfError::validation(format!( + "plugin '{}' ranks `next` by the rank of a type only [ranking.list] orders ({}): that rank \ + is computed over the type's live items, which the store-free lanes do not hold — answer \ + `next` from a store", + cfg.name, + reached.join("; ") + )) } /// Depth-first post-order over the type chain: every order a node reads the rank of lands before @@ -381,7 +481,146 @@ struct Context<'a> { ranks: HashMap>, } -impl Context<'_> { +impl<'a> Context<'a> { + /// A context over `subjects`, nothing read yet. + fn new(cfg: &'a PluginConfig, subjects: HashMap<&'a str, &'a ItemRow>) -> Context<'a> { + Context { + cfg, + subjects, + out: Adjacency::new(), + inb: Adjacency::new(), + mentioners: BTreeMap::new(), + neighbours: HashMap::new(), + labels: BTreeMap::new(), + custom: BTreeMap::new(), + ranks: HashMap::new(), + } + } + + /// Rank the chain bottom-up — each order its scope once, its dependents look the rank up — and + /// then the ranked rows' values. + fn finish( + mut self, + plan: &Plan<'_>, + verb: Verb, + ranked: &[ItemRow], + node_rows: &[Vec<&ItemRow>], + ) -> Derived { + let ranking = &self.cfg.ranking; + for (node, rows) in plan.chain.iter().zip(node_rows) { + let keys = &ranking.spec(node.0).by_type[&node.1].order; + let values: Vec> = + rows.iter().map(|r| self.values(keys, node.0, r)).collect(); + let mut order: Vec = (0..rows.len()).collect(); + let cmp = |a: usize, b: usize| { + super::keys_cmp_values(rows[a], rows[b], keys, Some(&values[a]), Some(&values[b])) + }; + order.sort_by(|a, b| cmp(*a, *b).then_with(|| rows[*a].id.cmp(&rows[*b].id))); + // A DENSE rank over the declared keys (spec §4): rows equal on every key share a rank, + // and the next distinct row takes the next integer. The id orders the sort, never the + // rank. + let mut ranks: HashMap = HashMap::with_capacity(rows.len()); + let mut rank = 0_i64; + for (n, &i) in order.iter().enumerate() { + if n > 0 && cmp(order[n - 1], i) != Ordering::Equal { + rank += 1; + } + ranks.insert(rows[i].id.clone(), rank); + } + self.ranks.insert(node.clone(), ranks); + } + let values = ranked + .iter() + .map(|r| { + let keys = plan.spec.order_for(r.item_type.as_deref()); + (r.id.clone(), self.values(keys, verb, r)) + }) + .collect(); + Derived { values } + } + + /// The reads from a [`RankingGraph`] instead of a store (6j6v.1c5s): the same adjacency, reverse + /// mentions, neighbour rows, labels and custom values [`read_edges`](Self::read_edges) reads, + /// taken from the graph's edges and from the board's tickets and the graph's neighbours. The + /// graph holds every edge of the kinds read that touches an active ticket with both ends live, + /// `parent` as the `present_parent` projection — the edges the store's pass yields for subjects + /// that are all active tickets, as `next`'s candidates are. + fn read_graph(&mut self, board: &ActiveBoard, graph: &RankingGraph, needs: &Needs) { + let mut mention_sources: BTreeMap> = BTreeMap::new(); + for e in &graph.edges { + if needs.kinds.contains(&e.kind) { + if self.subjects.contains_key(e.to.as_str()) { + self.inb + .entry((e.kind, e.to.clone())) + .or_default() + .push(e.from.clone()); + } + if self.subjects.contains_key(e.from.as_str()) { + self.out + .entry((e.kind, e.from.clone())) + .or_default() + .push(e.to.clone()); + } + } + if needs.mentioned_by + && e.kind == EdgeKind::Mentions + && self.subjects.contains_key(e.to.as_str()) + { + mention_sources + .entry(e.to.clone()) + .or_default() + .insert(e.from.as_str()); + } + } + // A row of the board: an active ticket's, or a live neighbour's. + let board_row = |id: &str| -> Option<&ItemRow> { + board + .tickets + .get(id) + .map(|t| &t.item) + .or_else(|| graph.neighbours.get(id).map(|n| &n.item)) + }; + for (target, froms) in mention_sources { + let rows: Vec = froms + .into_iter() + .filter_map(|f| board_row(f).cloned()) + .collect(); + if !rows.is_empty() { + self.mentioners.insert(target, rows); + } + } + let missing: BTreeSet<&str> = self + .out + .values() + .chain(self.inb.values()) + .flatten() + .map(String::as_str) + .filter(|id| !self.subjects.contains_key(id)) + .collect(); + self.neighbours = missing + .into_iter() + .filter_map(|id| board_row(id).map(|r| (id.to_string(), r.clone()))) + .collect(); + if needs.labels || needs.custom { + let mut ids: BTreeSet<&str> = self.neighbours.keys().map(String::as_str).collect(); + ids.extend(self.subjects.keys().copied()); + ids.extend(self.mentioners.values().flatten().map(|i| i.id.as_str())); + for id in ids { + let (labels, custom) = match (board.tickets.get(id), graph.neighbours.get(id)) { + (Some(t), _) => (&t.labels, &t.custom), + (None, Some(n)) => (&n.labels, &n.custom), + (None, None) => continue, + }; + if needs.labels && !labels.is_empty() { + self.labels.insert(id.to_string(), labels.clone()); + } + if needs.custom && !custom.is_empty() { + self.custom.insert(id.to_string(), custom.clone()); + } + } + } + } + /// The reads: the edge pass, the reverse mentions, the neighbours' rows, labels and custom /// fields — each ONE bulk read (per 500 ids where the store chunks), whatever the row count. fn read_edges(&mut self, store: &Store, needs: &Needs) -> Result<()> { diff --git a/crates/facade/tests/derived_ranking.rs b/crates/facade/tests/derived_ranking.rs index ae831c4..cdb778c 100644 --- a/crates/facade/tests/derived_ranking.rs +++ b/crates/facade/tests/derived_ranking.rs @@ -359,8 +359,37 @@ fn the_rank_follows_the_board_on_the_next_query() { } #[test] -fn the_store_free_lanes_refuse_a_plugin_with_derived_next_keys() { - // The server path is its own cut (1c5s); ranking without the edges would differ in silence. +fn the_store_free_next_ranks_from_the_ranking_graph_as_the_store_does() { + // 6j6v.1c5s: the board a server reads carries the graph the keys read, and the store-free + // `next` ranks from it exactly as the store path does — the spec's three-type chain. + let tmp = workspace("derived-rank-fixture"); + spec_board(tmp.path()); + let cfg = plugin::load("derived-rank-fixture").unwrap(); + let s = writer(tmp.path()); + let reads = read::next_ranking_reads(&cfg) + .unwrap() + .expect("derived next keys"); + assert_eq!( + reads.kinds.iter().copied().collect::>(), + [EdgeKind::ContributesTo, EdgeKind::Parent], + "the edges the next orders follow or count" + ); + assert!(reads.custom && !reads.labels, "{reads:?}"); + let board = read::with_ranking(&s, read::active_board(&s).unwrap(), &reads).unwrap(); + let got = read::next_active(&cfg, &board, NOW).unwrap(); + assert_eq!(ids(&got), ids(&read::next(&cfg, &s, NOW, None).unwrap())); + assert_eq!(of_type(&got, "project"), PROJECTS); + assert_eq!(of_type(&got, "action"), ACTIONS); + assert_eq!( + read::next_active_value(&cfg, &board, NOW).unwrap(), + read::next_to_value_with_custom(&cfg, &s, &read::next(&cfg, &s, NOW, None).unwrap()) + .unwrap() + ); +} + +#[test] +fn the_store_free_next_refuses_a_board_read_without_the_graph_its_keys_read() { + // Ranking without the edges would order apart from the store path in silence. let tmp = workspace("derived-rank-fixture"); spec_board(tmp.path()); let cfg = plugin::load("derived-rank-fixture").unwrap(); @@ -369,6 +398,180 @@ fn the_store_free_lanes_refuse_a_plugin_with_derived_next_keys() { let err = read::next_active(&cfg, &board, NOW).unwrap_err(); assert_eq!(err.kind, ErrorKind::Validation); assert!(err.msg.contains("derived ranking keys"), "{}", err.msg); + // A graph read for less than the keys need is refused too: here without the custom values + // the goals' `order` lives in. + let mut reads = read::next_ranking_reads(&cfg).unwrap().unwrap(); + reads.custom = false; + let short = read::with_ranking(&s, read::active_board(&s).unwrap(), &reads).unwrap(); + let err = read::next_active(&cfg, &short, NOW).unwrap_err(); + assert_eq!(err.kind, ErrorKind::Validation); + // `blocked` and `deferred` rank by no derived key and need no graph. + read::blocked_active(&cfg, &board); + read::deferred_active(&cfg, &board, NOW).unwrap(); +} + +/// A plugin ranking `next` by its neighbours with no status filter, so closed, archived and +/// deleted neighbours each decide something (owner, 2026-10-11: a key reads every LIVE neighbour). +const LIVE_ORDERS: &str = r#" +name = "live-neighbours-fixture" +[description] +en = "x" +de = "y" +[priority] +labels = ["P0", "P1", "P2", "P3"] +[types] +list = ["goal", "project", "action"] +[vocabulary.status] +open = "open" +in_progress = "in progress" +closed = "closed" +[ranking.next] +types = ["project", "action"] +order = [ { field = "priority", dir = "asc" } ] +[ranking.next.project] +order = [ + { related = "contributes_to", type = "goal", of = { field = "order" }, pick = "min", nulls = "last" }, + { count = "children", dir = "desc" }, + { count = "mentioned_by", dir = "desc" }, +] +[ranking.next.action] +order = [ { related = "parent", type = "project", of = "rank", nulls = "last" } ] +[presentation.list] +columns = ["id"] +[fields.order] +type = "text" +on = ["goal"] +"#; + +#[test] +fn a_closed_archived_or_deleted_neighbour_reads_alike_with_and_without_a_store() { + // Goals: `gc` closed (order 1), `ga` archived (order 2), `go` open (order 3), `gd` deleted + // (order 0). Projects `pa`..`pd` contribute to one each: a closed or archived goal still gives + // its order, a deleted one gives none (null, last). Projects `q1`..`q4` have no goal: `q1` has + // three children — two closed, one archived — `q2` one open child, `q3` is mentioned by a + // closed project and `q4` by nothing. Every child and mentioner but `q2`'s is out of `next`. + let cfg: plugin::PluginConfig = toml::from_str(LIVE_ORDERS).unwrap(); + let mut s = Store::open_in_memory(1); + for (g, order) in [ + ("zz12.gc", "1"), + ("zz12.ga", "2"), + ("zz12.go", "3"), + ("zz12.gd", "0"), + ] { + item(&mut s, g, "goal", "1"); + custom(&mut s, g, "order", order); + } + close(&mut s, "zz12.gc"); + s.set_field("zz12.ga", "archived", Some("2026-06-01".into()), "t"); + for (p, g) in [ + ("zz12.pa", "zz12.gc"), + ("zz12.pb", "zz12.ga"), + ("zz12.pc", "zz12.gd"), + ("zz12.pd", "zz12.go"), + ] { + item(&mut s, p, "project", "1"); + edge(&mut s, p, g, EdgeKind::ContributesTo); + } + s.delete_item("zz12.gd", "t"); + for q in ["zz12.q1", "zz12.q2", "zz12.q3", "zz12.q4"] { + item(&mut s, q, "project", "1"); + } + for (a, parent) in [ + ("zz12.a1", "zz12.q1"), + ("zz12.a2", "zz12.q1"), + ("zz12.a3", "zz12.q1"), + ("zz12.a4", "zz12.q2"), + ] { + item(&mut s, a, "action", "1"); + s.set_parent(a, parent, "t").unwrap(); + } + close(&mut s, "zz12.a1"); + close(&mut s, "zz12.a2"); + s.set_field("zz12.a3", "archived", Some("2026-06-01".into()), "t"); + item(&mut s, "zz12.px", "project", "1"); + edge(&mut s, "zz12.px", "zz12.q3", EdgeKind::Mentions); + close(&mut s, "zz12.px"); + + let want = [ + "zz12.pa", "zz12.pb", "zz12.pd", "zz12.q1", "zz12.q2", "zz12.q3", "zz12.pc", "zz12.q4", + ]; + let store = read::next(&cfg, &s, NOW, None).unwrap(); + assert_eq!(of_type(&store, "project"), want, "the store path"); + let reads = read::next_ranking_reads(&cfg).unwrap().unwrap(); + let board = read::with_ranking(&s, read::active_board(&s).unwrap(), &reads).unwrap(); + let graph = board.ranking.as_ref().unwrap(); + for id in ["zz12.gc", "zz12.ga", "zz12.a1", "zz12.a3", "zz12.px"] { + assert!( + graph.neighbours.contains_key(id), + "{id} is a live neighbour" + ); + } + assert!( + !graph.neighbours.contains_key("zz12.gd"), + "a deleted one is not" + ); + assert!( + graph.edges.iter().all(|e| e.to != "zz12.gd"), + "nor an edge to it" + ); + assert_eq!( + ids(&read::next_active(&cfg, &board, NOW).unwrap()), + ids(&store) + ); +} + +#[test] +fn a_next_order_reading_the_rank_of_a_type_only_list_orders_is_refused_without_a_store() { + // Spec §12 point 1 lets a `next` key read the rank `[ranking.list]` gives a type. That rank is + // computed over the type's live items, which a board of active tickets does not hold: the + // store-free `next` refuses the plugin (owner, 2026-10-11), the store path ranks it. Any other + // derived `next` key ranks without a store. + let cfg: plugin::PluginConfig = toml::from_str( + r#" + name = "list-rank-fixture" + [description] + en = "x" + de = "y" + [priority] + labels = ["P0", "P1"] + [types] + list = ["action", "person"] + [vocabulary.status] + open = "open" + in_progress = "in progress" + closed = "closed" + [ranking.next] + order = [ { field = "priority" } ] + [ranking.next.action] + order = [ { related = "mentions", type = "person", of = "rank", nulls = "last" } ] + [ranking.list.person] + order = [ { field = "priority", dir = "asc" } ] + [presentation.list] + columns = ["id"] + "#, + ) + .unwrap(); + let mut s = Store::open_in_memory(1); + item(&mut s, "zz12.a1", "action", "1"); + item(&mut s, "zz12.u1", "person", "0"); + edge(&mut s, "zz12.a1", "zz12.u1", EdgeKind::Mentions); + let err = read::next_ranking_reads(&cfg).unwrap_err(); + assert_eq!(err.kind, ErrorKind::Validation); + assert!( + err.msg + .contains("[ranking.next.action] reads the rank of 'person'"), + "{}", + err.msg + ); + let board = read::active_board(&s).unwrap(); + assert_eq!( + read::next_active(&cfg, &board, NOW).unwrap_err().kind, + ErrorKind::Validation + ); + assert!( + read::next(&cfg, &s, NOW, None).is_ok(), + "the store path ranks it" + ); } // ---- each kind over hand-built boards (§9 "per kind") ------------------------------------- diff --git a/crates/fold-ddb/Cargo.toml b/crates/fold-ddb/Cargo.toml index 97c413c..27dc171 100644 --- a/crates/fold-ddb/Cargo.toml +++ b/crates/fold-ddb/Cargo.toml @@ -37,3 +37,6 @@ aws-sdk-dynamodb = { version = "1", optional = true } tokio = { version = "1", features = ["rt-multi-thread", "macros", "time"] } # The records differential test (6j6v.t1ym) parses a plugin fixture that declares a custom field. toml = "0.8" +# The derived-keys differential test (6j6v.1c5s) registers its plugin fixture, so `plugin::load` +# validates it as it validates a consumer's plugin. +inventory = "0.3" diff --git a/crates/fold-ddb/src/board.rs b/crates/fold-ddb/src/board.rs index 2ffc850..36aea0e 100644 --- a/crates/fold-ddb/src/board.rs +++ b/crates/fold-ddb/src/board.rs @@ -49,6 +49,19 @@ //! Only `contributes_to` adds get one: folding another edge's add costs nothing more, and folding //! an edge remove one `GetItem` of its add, to learn its kind. //! +//! # What an adjacency entry says +//! +//! Revision 3 (6j6v.1c5s) makes the `.adj##` entries self-describing, as the entries +//! under a ticket are: each copies the edge add's `from_id`, `to_id`, `kind`, `lamport` and `site` +//! and says whether the add is removed ([`REMOVED`]). Every edge add has one under each end, +//! whatever its kind and whatever the ends' activity, so ONE Query on a ticket's `.adj#` prefix +//! reads every edge touching it, in and out, with its presence — what the derived ranking keys read +//! ([`crate::records::with_ranking`]): inbound `contributes_to`, `mentions` both ways, the +//! `parent` edges a child's current parent is picked from. The entries keep their place and key; +//! only their attributes grow. Folding an edge add reads its remove (a `dep` or `parent` add read it +//! already, for its flag); folding an edge remove now reads the add and the remove and rewrites +//! both entries — two `GetItem`s and two writes, one when the edge is a loop. +//! //! # One writer per stream //! //! The rows a reducer describes converge whatever the order and however many folders write them. @@ -279,7 +292,7 @@ pub(crate) async fn after(table: &T, changes: &[Change]) -> Result<(), link_edge(table, tag).await?; } for tag in removed { - refresh_edge(table, tag).await?; + link_edge(table, tag).await?; } for (at, tag) in owned_tags { refresh_owned(table, &OWNED[at], tag).await?; @@ -304,7 +317,7 @@ pub(crate) async fn refresh_item(table: &T, id: &str) -> Result<(), T: for entry in table.query_prefix(&adjacency_prefix(id)).await? { if let Some(tag) = text(&entry, "tag") { if let Some(edge) = table.get(&tag_key("edge_adds", tag)).await? { - flag_edge(table, tag, edge, Some((id, active))).await?; + flag_edge(table, tag, edge, None, Some((id, active))).await?; } } } @@ -349,46 +362,68 @@ pub(crate) async fn refresh_item(table: &T, id: &str) -> Result<(), T: Ok(()) } -/// Record that edge `tag` touches its ends, then set its flag. -pub(crate) async fn link_edge(table: &T, tag: &str) -> Result<(), T::Error> { - let Some(edge) = table.get(&tag_key("edge_adds", tag)).await? else { - return Ok(()); - }; - let ends: BTreeSet<&str> = [text(&edge, "from_id"), text(&edge, "to_id")] +/// The cells of edge add `edge` an adjacency entry copies beside its tag and [`REMOVED`] (index +/// revision 3, 6j6v.1c5s): its ends, its kind and its coordinate. +const EDGE_COPIES: [&str; 5] = ["from_id", "to_id", "kind", "lamport", "site"]; + +/// The adjacency entries of edge add `tag` — one under each end, the same entry twice for an edge +/// from a ticket to itself — from the add's row and whether it is removed. +async fn write_adjacency( + table: &T, + tag: &str, + edge: &Row, + removed: bool, +) -> Result<(), T::Error> { + let mut set = vec![("tag".to_string(), Cell::Text(tag.to_string()))]; + for column in EDGE_COPIES { + match edge.get(column) { + None | Some(Cell::Null) => {} + Some(cell) => set.push((column.to_string(), cell.clone())), + } + } + set.push((REMOVED.to_string(), Cell::Int(i64::from(removed)))); + let ends: BTreeSet<&str> = [text(edge, "from_id"), text(edge, "to_id")] .into_iter() .flatten() .collect(); for end in ends { table - .write(&Write::put( - adjacency_key(end, tag), - vec![("tag".to_string(), Cell::Text(tag.to_string()))], - )) + .write(&Write::put(adjacency_key(end, tag), set.clone())) .await?; } - flag_edge(table, tag, edge, None).await + Ok(()) } -/// Set edge `tag`'s flag from its row, its remove and its ends. -pub(crate) async fn refresh_edge(table: &T, tag: &str) -> Result<(), T::Error> { - match table.get(&tag_key("edge_adds", tag)).await? { - Some(edge) => flag_edge(table, tag, edge, None).await, - None => Ok(()), - } +/// Record that edge `tag` touches its ends — the entry under each end says what the edge is and +/// whether it is removed — then set its flag. Run when the add is folded and again when its remove +/// is; nothing while the add has not been folded, since the add writes both. +pub(crate) async fn link_edge(table: &T, tag: &str) -> Result<(), T::Error> { + let Some(edge) = table.get(&tag_key("edge_adds", tag)).await? else { + return Ok(()); + }; + let removed = table.get(&tag_key("edge_removes", tag)).await?.is_some(); + write_adjacency(table, tag, &edge, removed).await?; + flag_edge(table, tag, edge, Some(removed), None).await } -/// `known` is a ticket whose activity the caller knows ahead of its stored flag. +/// `removed` is whether the edge's remove is folded, when the caller has read it already; `known` +/// a ticket whose activity the caller knows ahead of its stored flag. async fn flag_edge( table: &T, tag: &str, edge: Row, + removed: Option, known: Option<(&str, bool)>, ) -> Result<(), T::Error> { let indexed = matches!( text(&edge, "kind").and_then(EdgeKind::parse), Some(EdgeKind::Dep | EdgeKind::Parent) ); - let mut flag = indexed && table.get(&tag_key("edge_removes", tag)).await?.is_none(); + let mut flag = indexed + && match removed { + Some(removed) => !removed, + None => table.get(&tag_key("edge_removes", tag)).await?.is_none(), + }; if flag { let mut any_active = false; for end in [text(&edge, "from_id"), text(&edge, "to_id")] diff --git a/crates/fold-ddb/src/fold.rs b/crates/fold-ddb/src/fold.rs index 6d0c283..e4f8075 100644 --- a/crates/fold-ddb/src/fold.rs +++ b/crates/fold-ddb/src/fold.rs @@ -21,7 +21,9 @@ //! bumps it, and every stream folded before is no longer current and is folded again the same way. //! Revision 1 (6j6v.t1ym) adds the `.ladj#`/`.tadj#` entries; a stream folded before it has none, //! so its watermark names no index revision at all and the next run refolds it. Revision 2 -//! (6j6v.15ed) adds the `.cadj#` entries, a ticket's outgoing `contributes_to` edges. +//! (6j6v.15ed) adds the `.cadj#` entries, a ticket's outgoing `contributes_to` edges. Revision 3 +//! (6j6v.1c5s) makes the `.adj#` entries self-describing — ends, kind, coordinate, presence — for +//! the derived ranking keys ([`crate::records::with_ranking`]). //! //! # What the server cannot see //! @@ -46,7 +48,9 @@ pub const INDEX_DOMAIN: &str = "index"; /// The revision of those rules. 1: the entries under a ticket for its label and thread link adds /// (6j6v.t1ym). 2: the entries under a ticket for its outgoing `contributes_to` edges (6j6v.15ed). -pub const INDEX_REVISION: i64 = 2; +/// 3: the adjacency entries copy their edge's ends, kind and coordinate and say whether it is +/// removed (6j6v.1c5s). +pub const INDEX_REVISION: i64 = 3; /// The platform's reducers — the board's, the chat's and the memory's. pub fn platform_reducers() -> Vec> { diff --git a/crates/fold-ddb/src/lib.rs b/crates/fold-ddb/src/lib.rs index 868b37b..f97134f 100644 --- a/crates/fold-ddb/src/lib.rs +++ b/crates/fold-ddb/src/lib.rs @@ -23,6 +23,10 @@ //! | GSI `active` | `nxf_active` / `sk` | S/S | sparse: active tickets and the present dep/parent edges touching one | //! | GSI `dated` | `nxf_dated` / `nxf_dated_at` | S/S | sparse: `#closed` and `#archived`, by `␟` | //! +//! The `.adj##` entries are plain rows in the stream's partition, on no index. Since +//! index revision 3 (6j6v.1c5s) each also carries its edge's `from_id`, `to_id`, `kind`, `lamport`, +//! `site` and a `removed` flag — attributes, not keys: the contract above is unchanged. +//! //! Both indexes project ALL attributes, so a lane is one Query and no follow-up reads. The lanes //! need nothing else; the RECORDS a lane is shown with carry joins from other tables, which //! [`records`] reads with a fixed number of requests per active ticket. Billing mode @@ -61,7 +65,10 @@ //! one write. A label or thread link add, or its remove, adds two `GetItem`s and at most one write //! for the entry under the ticket (`board`, "Who owns a label or a thread link"). Reading the //! active tickets' records costs `4n + 1` requests for `n` tickets, plus one per parent outside -//! the selection, at most [`records::READ_CONCURRENCY`] tickets at a time ([`records`]). No request +//! the selection, at most [`records::READ_CONCURRENCY`] tickets at a time ([`records`]). Folding an +//! edge remove reads the add and the remove and rewrites the entry under each end. The graph the +//! derived ranking keys read ([`records::with_ranking`]) costs one Query per active ticket and, per +//! inactive neighbour, one `GetItem` plus up to three Queries. No request //! is a Scan: every read names its partition key, the board's through an index. The //! `dynamodb-local` CI job counts the requests of a fold and of a records read, and proves it. //! diff --git a/crates/fold-ddb/src/records.rs b/crates/fold-ddb/src/records.rs index cb930d8..4d52e05 100644 --- a/crates/fold-ddb/src/records.rs +++ b/crates/fold-ddb/src/records.rs @@ -34,6 +34,11 @@ //! a time, so a board's latency is about `n / READ_CONCURRENCY` round trips, and no board sends more //! than that many requests at once. //! +//! One more is read only for a plugin that ranks `next` by derived ranking keys: +//! [`with_ranking`] fills the board's [`RankingGraph`] — the edges around the active tickets and +//! their live neighbours — from the adjacency entries (index revision 3, 6j6v.1c5s). Its cost is in +//! its own doc. +//! //! # Two points in time //! //! The selection is an index read, eventually consistent; every join is a row read, strongly @@ -47,7 +52,8 @@ //! # A stream folded by older rules //! //! The entries under a ticket came with index revision 1, the `.cadj#` entries with revision 2 -//! (6j6v.15ed). A stream whose watermark names an older +//! (6j6v.15ed), the self-describing `.adj#` entries with revision 3 (6j6v.1c5s). A stream whose +//! watermark names an older //! one — or none, folded before revisions were recorded for the index — has no entries, and its //! records would come back without labels or conversations. [`active_board`] refuses such a stream //! ([`RecordsError::NotCurrent`]); the fold run refolds it ([`Folder::is_current`]), after which it @@ -58,12 +64,16 @@ use crate::board::{int, item_key, Selection, REMOVED}; use crate::fold::{Folder, INDEX_DOMAIN, INDEX_REVISION}; use crate::layout::{ - contributes_adjacency_prefix, label_adjacency_prefix, link_adjacency_prefix, row_key, + adjacency_prefix, contributes_adjacency_prefix, label_adjacency_prefix, link_adjacency_prefix, + row_key, }; use crate::table::{text, Row, Table}; use futures_util::stream::{self, StreamExt, TryStreamExt}; -use nexus_flow_core::model::{ItemRow, LinkWeight}; -use nexus_flow_facade::read::{ActiveBoard, ActiveTicket, ParentRef}; +use nexus_flow_core::graph::Edge; +use nexus_flow_core::model::{EdgeKind, ItemRow, LinkWeight}; +use nexus_flow_facade::read::{ + ActiveBoard, ActiveTicket, Neighbour, ParentRef, RankingGraph, RankingReads, +}; use nxs_foundation::change::Cell; use std::collections::{BTreeMap, BTreeSet}; @@ -206,6 +216,164 @@ async fn contributes_to(table: &T, id: &str) -> Result, T: Ok(targets.into_iter().collect()) } +/// One present edge as its adjacency entry tells it (index revision 3): the add's ends, kind and +/// coordinate (`-1` where the add has none, as `present_parent` reads it). +#[derive(Debug, Clone)] +struct EdgeEntry { + from: String, + to: String, + kind: EdgeKind, + lamport: i64, + site: i64, +} + +/// The present edges touching ticket `id`, by tag: one Query on its `.adj#` prefix. +async fn present_edges( + table: &T, + id: &str, +) -> Result, T::Error> { + Ok(present_entries(table, &adjacency_prefix(id)) + .await? + .iter() + .filter_map(|e| { + let entry = EdgeEntry { + from: text(e, "from_id")?.to_string(), + to: text(e, "to_id")?.to_string(), + kind: text(e, "kind").and_then(EdgeKind::parse)?, + lamport: int(e, "lamport").unwrap_or(-1), + site: int(e, "site").unwrap_or(-1), + }; + Some((text(e, "tag")?.to_string(), entry)) + }) + .collect()) +} + +/// The current parent of `id` among the present edges touching it — `present_parent`'s pick: of its +/// present outgoing `parent` edges, the one with the highest `(lamport, site, to_id)`. +fn current_parent(id: &str, edges: &BTreeMap) -> Option { + edges + .values() + .filter(|e| e.kind == EdgeKind::Parent && e.from == id) + .max_by(|a, b| (a.lamport, a.site, &a.to).cmp(&(b.lamport, b.site, &b.to))) + .map(|e| e.to.clone()) +} + +/// `board` with its [`RankingGraph`] for `reads` (6j6v.1c5s): the graph the derived ranking keys of +/// `[ranking.next]` read, from the folded tables — what [`read::with_ranking`] reads from a store, +/// edge for edge and neighbour for neighbour. A server calls it when +/// [`read::next_ranking_reads`] names reads for its plugin, so a plugin without derived `next` keys +/// pays nothing for it. The stream must be current, as for [`active_board`]: the entries read here +/// came with index revision 3. +/// +/// | read | per | +/// |-----------------------------------------------|-------------------------------------------| +/// | one Query on `.adj##` | active ticket | +/// | one `GetItem` of its row | other end of such an edge that is not active | +/// | one Query on `.adj##` (its parent) | such end that is live | +/// | one Query on `.ladj##`, `custom_fields##` | such end, when `reads` asks for labels, custom values | +/// +/// So `n` active tickets with `p` distinct inactive neighbours, `q` of them live, cost `n + q` +/// Queries and `p` `GetItem`s — `n + 3q` Queries with labels and custom values — at most +/// [`READ_CONCURRENCY`] items at a time, and nothing grows with an item's history: an entry says +/// whether its add is removed. Every edge touching an active ticket is read once, from one end's +/// entries, whatever the plugin's keys — the edge pass the store path runs once per query. +/// +/// [`read::with_ranking`]: nexus_flow_facade::read::with_ranking +/// [`read::next_ranking_reads`]: nexus_flow_facade::read::next_ranking_reads +pub async fn with_ranking( + table: &T, + mut board: ActiveBoard, + reads: &RankingReads, +) -> Result> { + // Every present edge of a kind read that touches an active ticket, once per tag. + let ids: Vec = board.tickets.keys().cloned().collect(); + let per_ticket: Vec> = stream::iter(ids) + .map(|id| async move { present_edges(table, &id).await }) + .buffered(READ_CONCURRENCY) + .try_collect() + .await?; + let mut touching: BTreeMap = BTreeMap::new(); + for edges in per_ticket { + touching.extend( + edges + .into_iter() + .filter(|(_, e)| reads.kinds.contains(&e.kind)), + ); + } + + // The other ends that are not active: their rows, and for the live ones their current parent + // and the joins asked for. + let others: BTreeSet = touching + .values() + .flat_map(|e| [&e.from, &e.to]) + .filter(|id| !board.tickets.contains_key(*id)) + .cloned() + .collect(); + let rows: Vec<(String, Option)> = stream::iter(others) + .map(|id| async move { + let row = table.get(&item_key(&id)).await?; + Ok::<_, T::Error>((id, row)) + }) + .buffered(READ_CONCURRENCY) + .try_collect() + .await?; + let live = rows + .into_iter() + .filter_map(|(id, row)| row.map(|r| (id, r))) + .filter(|(_, r)| text(r, "deleted") != Some("1")); + let neighbours: BTreeMap = stream::iter(live) + .map(|(id, row)| async move { + let parent = current_parent(&id, &present_edges(table, &id).await?); + let mut n = Neighbour::new(item_row(&row, parent)); + if reads.labels { + n.labels = labels(table, &id).await?; + } + if reads.custom { + n.custom = custom(table, &id).await?; + } + Ok::<_, T::Error>((id, n)) + }) + .buffered(READ_CONCURRENCY) + .try_collect() + .await?; + + // The edges between live ends, a `parent` edge only from a child to its current parent. + let parent_of = |id: &str| -> Option<&str> { + board + .tickets + .get(id) + .map(|t| &t.item) + .or_else(|| neighbours.get(id).map(|n| &n.item)) + .and_then(|i| i.belongs_to.as_deref()) + }; + let live_end = |id: &str| board.tickets.contains_key(id) || neighbours.contains_key(id); + let edges: BTreeSet = touching + .into_values() + .filter(|e| live_end(&e.from) && live_end(&e.to)) + .filter(|e| e.kind != EdgeKind::Parent || parent_of(&e.from) == Some(e.to.as_str())) + .map(|e| Edge { + from: e.from, + to: e.to, + kind: e.kind, + }) + .collect(); + // A child whose current parent is elsewhere was read for its parent edge only, which is not in + // the graph: it is no neighbour. + let ends: BTreeSet<&str> = edges + .iter() + .flat_map(|e| [e.from.as_str(), e.to.as_str()]) + .collect(); + let neighbours = neighbours + .into_iter() + .filter(|(id, _)| ends.contains(id.as_str())) + .collect(); + let mut graph = RankingGraph::new(reads.clone()); + graph.edges = edges; + graph.neighbours = neighbours; + board.ranking = Some(graph); + Ok(board) +} + /// An `items` row as the store reads it, with the ticket's `belongs_to` (the `present_parent` join). pub(crate) fn item_row(row: &Row, belongs_to: Option) -> ItemRow { let cell = |c: &str| text(row, c).map(str::to_string); diff --git a/crates/fold-ddb/tests/derived_differential.rs b/crates/fold-ddb/tests/derived_differential.rs new file mode 100644 index 0000000..cd5b228 --- /dev/null +++ b/crates/fold-ddb/tests/derived_differential.rs @@ -0,0 +1,483 @@ +//! A server ranks `next` by derived ranking keys as a replica does (6j6v.1c5s, +//! docs/specs/derived-ranking-keys.md §6 and §9 "differential"). +//! +//! Generated boards of three types — goals, projects, actions — written by two replicas apart and +//! then merged: every status, archived and deleted items, priorities, the goals' `order` custom +//! field, `waiting` labels, and `parent`, `contributes_to`, `mentions` and `dep` edges with removes. +//! Two replicas parent the same child apart, so a child can have two present parent edges and only +//! `present_parent`'s pick may count. Each board is folded into a [`MemTable`] in a shuffled order +//! with duplicates, as the relay hands it over. +//! +//! Then three paths rank `next` under a plugin whose orders hold every derived key kind, each +//! reading closed, archived and deleted neighbours, down a three-type chain (action → project → +//! goal): +//! +//! 1. the SQL path: `read::next` over the store; +//! 2. the Board path: `read::next_active` over the store's own [`read::ActiveBoard`] and +//! [`read::RankingGraph`] (`read::active_board` + `read::with_ranking`); +//! 3. the fold-ddb path: `read::next_active` over the folded tables' (`board::select` + +//! `records::active_board` + `records::with_ranking`). +//! +//! All three must give the same rows in the same order and the same JSON, and the two inputs must +//! be equal, edge for edge and neighbour for neighbour. Folding the same log in another order must +//! read the same graph (determinism). + +use nexus_flow_core::model::{EdgeKind, ItemRow, MergeStrategy}; +use nexus_flow_core::store::Store; +use nexus_flow_facade::plugin::{self, PluginConfig, PluginRegistration}; +use nexus_flow_facade::read; +use nxs_fold_ddb::fold::{Folder, INDEX_DOMAIN}; +use nxs_fold_ddb::layout::WATERMARK_KEY; +use nxs_fold_ddb::mem::{ready, MemTable}; +use nxs_fold_ddb::table::Table; +use nxs_fold_ddb::write::{Cond, Write}; +use nxs_fold_ddb::{board, records}; +use nxs_foundation::change::Cell; +use std::collections::{BTreeMap, BTreeSet}; + +mod common; +use common::{delivered, Lcg, NOW}; + +/// Every derived key kind, each able to read a neighbour that is not active: a goal's order and +/// rank from a closed or archived goal, counts without a status filter, a `dep` blocker's status +/// (only a closed blocker leaves its dependent in `next`), the `belongs_to` of a mentioned item. +const ORDERS: &str = r#" +name = "derived-differential-fixture" +[description] +en = "x" +de = "y" +[priority] +labels = ["P0", "P1", "P2", "P3", "P4"] +[types] +list = ["goal", "project", "action"] +[vocabulary.status] +open = "open" +in_progress = "in progress" +closed = "closed" +[ranking.next] +types = ["action", "project", "goal"] +order = [ { field = "priority", dir = "asc" } ] +[ranking.next.goal] +order = [ + { count = "contributors", type = "project", dir = "desc" }, + { field = "priority", dir = "asc" }, +] +[ranking.next.project] +order = [ + { related = "contributes_to", type = "goal", of = { field = "order" }, pick = "min", nulls = "last" }, + { related = "parent", type = "goal", of = "rank", pick = "max", nulls = "first" }, + { sum = [ { count = "mentioned_by" }, + { count = "mentioned_by", type = "action", label = "waiting", status = ["open", "in_progress"] } ], dir = "desc" }, + { count = "children", type = "action", dir = "desc" }, + { field = "priority", dir = "asc" }, +] +[ranking.next.action] +order = [ + { related = "parent", type = "project", of = "rank", nulls = "last" }, + { related = "dep", of = { field = "status", precedence = ["closed", "in_progress", "open"] }, pick = "min" }, + { count = "dependents", dir = "desc" }, + { related = "mentions", of = { field = "belongs_to" }, pick = "max", nulls = "last" }, + { count = "contributors", status = ["closed"], dir = "desc" }, + { field = "priority", dir = "asc" }, +] +[presentation.list] +columns = ["id"] +[fields.order] +type = "text" +on = ["goal"] +"#; + +inventory::submit!(PluginRegistration { + name: "derived-differential-fixture", + toml: ORDERS, + order: 140, +}); + +fn cfg() -> PluginConfig { + plugin::load("derived-differential-fixture").expect("the fixture is a valid plugin") +} + +/// The items of a generated board: three goals, five projects, eight actions. +fn items() -> Vec<(String, &'static str)> { + let mut out = Vec::new(); + for (ty, letter, n) in [("goal", 'g', 3), ("project", 'p', 5), ("action", 'a', 8)] { + for i in 0..n { + out.push((format!("ab12.{letter}{i:03}"), ty)); + } + } + out +} + +/// One replica's share of the writing. `added` holds the edges this replica added, so a remove +/// usually takes back a real one. +fn write(s: &mut Store, r: &mut Lcg, added: &mut Vec<(String, String, EdgeKind)>, steps: usize) { + let all = items(); + let pick = |r: &mut Lcg| all[r.next(all.len() as u64) as usize].0.clone(); + for step in 0..steps { + s.set_wall_clock(&format!( + "2026-10-0{}T{:02}:00:00Z", + 1 + r.next(7), + step % 24 + )); + let (x, y) = (pick(r), pick(r)); + match r.next(16) { + 0 => s.set_field(&x, "status", Some("in_progress".into()), "u"), + 1 | 2 => s.set_field(&x, "status", Some("closed".into()), "u"), + 3 => s.set_field(&x, "status", Some("open".into()), "u"), + 4 => s.set_field( + &x, + "archived", + Some(format!("2026-10-0{}", 1 + r.next(7))), + "u", + ), + 5 => s.set_field(&x, "archived", None, "u"), + 6 if r.next(3) == 0 => s.delete_item(&x, "u"), + 7 => s.set_field(&x, "priority", Some(r.next(5).to_string()), "u"), + 8..=12 => { + // Dependencies are rare: an active blocker takes its dependent out of `next`. + let kind = match r.next(10) { + 0..=2 => EdgeKind::Parent, + 3..=5 => EdgeKind::ContributesTo, + 6..=8 => EdgeKind::Mentions, + _ => EdgeKind::Dep, + }; + s.add_edge(&x, &y, kind, "u"); + added.push((x, y, kind)); + } + 13 if !added.is_empty() => { + let (from, to, kind) = added.swap_remove(r.next(added.len() as u64) as usize); + s.remove_edge(&from, &to, kind, "u"); + } + 14 => { + if r.next(3) == 0 { + s.remove_label(&x, "waiting", "u"); + } else { + s.add_label(&x, "waiting", "u"); + } + } + _ => s.set_custom_field_merge( + &x, + "order", + Some(r.next(6).to_string()), + "u", + MergeStrategy::Lww, + ), + } + } +} + +/// A merged board: two replicas that wrote apart, then exchanged everything. +fn gen_board(seed: u64) -> Store { + let mut r = Lcg(seed ^ 0xd1ff); + let mut a = Store::open_in_memory(1); + for (id, ty) in items() { + a.create_item(&id, ty, "T", "u"); + a.set_field(&id, "priority", Some(r.next(5).to_string()), "u"); + if ty == "goal" && r.next(4) != 0 { + a.set_custom_field_merge( + &id, + "order", + Some(r.next(6).to_string()), + "u", + MergeStrategy::Lww, + ); + } + } + let mut b = Store::open_in_memory(2); + b.apply(&a.export()); + let steps = 30 + r.next(40) as usize; + write(&mut a, &mut r, &mut Vec::new(), steps); + write(&mut b, &mut r, &mut Vec::new(), steps); + let from_b = b.export(); + a.apply(&from_b); + a +} + +/// `ops` folded into a fresh table. +fn fold(folder: &Folder, ops: &[(i64, nxs_foundation::model::Op)]) -> MemTable { + let t = MemTable::new("stream-1"); + ready(folder.fold_batch(&t, ops)).unwrap(); + t +} + +/// The served board with its ranking graph, as a server reads it for `reads` — and with the +/// contributes-to targets, so its tickets compare whole against `read::active_board`'s. +fn served(t: &MemTable, reads: &read::RankingReads) -> read::ActiveBoard { + let selection = ready(board::select(t)).unwrap(); + let plain = ready(records::active_board(t, selection)).unwrap(); + let plain = ready(records::with_contributes_to(t, plain)).unwrap(); + ready(records::with_ranking(t, plain, reads)).unwrap() +} + +fn ids(items: &[ItemRow]) -> Vec { + items.iter().map(|i| i.id.clone()).collect() +} + +fn count(s: &Store, sql: &str) -> usize { + s.connection() + .query_row(sql, [], |r| r.get::<_, i64>(0)) + .unwrap() as usize +} + +/// What the generated boards held, so the comparison is known to have compared something worth +/// comparing. +#[derive(Default)] +struct Seen(BTreeMap<&'static str, usize>); + +impl Seen { + fn add(&mut self, what: &'static str, n: usize) { + *self.0.entry(what).or_default() += n; + } + + fn assert_all(&self) { + for (what, n) in &self.0 { + assert!(*n > 0, "the generated boards have {what}: {:?}", self.0); + } + } +} + +#[test] +fn the_server_ranks_next_by_derived_keys_as_the_replica_does() { + let folder = Folder::platform().unwrap(); + let cfg = cfg(); + let reads = read::next_ranking_reads(&cfg) + .unwrap() + .expect("derived next keys"); + assert_eq!(reads.kinds.len(), 4, "every edge kind: {reads:?}"); + assert!(reads.labels && reads.custom, "{reads:?}"); + let mut seen = Seen::default(); + for seed in 0..60 { + let s = gen_board(seed); + let ctx = format!("seed {seed}"); + let t = fold(&folder, &delivered(&s, seed)); + + let local = read::with_ranking(&s, read::active_board(&s).unwrap(), &reads).unwrap(); + let server = served(&t, &reads); + assert_eq!(server.tickets, local.tickets, "active tickets, {ctx}"); + assert_eq!(server.ranking, local.ranking, "the ranking graph, {ctx}"); + + // 1 = 2 = 3: the SQL path, the Board path, the fold-ddb path. + let want = read::next(&cfg, &s, NOW, None).unwrap(); + assert_eq!( + ids(&read::next_active(&cfg, &local, NOW).unwrap()), + ids(&want), + "SQL path vs Board path, {ctx}" + ); + assert_eq!( + ids(&read::next_active(&cfg, &server, NOW).unwrap()), + ids(&want), + "SQL path vs fold-ddb path, {ctx}" + ); + assert_eq!( + read::next_active_value(&cfg, &server, NOW).unwrap(), + read::next_to_value_with_custom(&cfg, &s, &want).unwrap(), + "next JSON, {ctx}" + ); + + // Determinism: the same log folded in another order reads the same graph (every fourth + // board — a fold is most of a seed's time). + if seed % 4 == 0 { + let again = served(&fold(&folder, &delivered(&s, seed ^ 0xffff)), &reads); + assert_eq!(again.ranking, server.ranking, "another fold order, {ctx}"); + } + + // What the boards held. + let graph = server.ranking.as_ref().unwrap(); + let candidates: BTreeSet<&str> = want.iter().map(|i| i.id.as_str()).collect(); + seen.add("next rows", want.len()); + for n in graph.neighbours.values() { + let archived = n.item.archived.is_some(); + seen.add( + "closed neighbours", + usize::from(!archived && n.item.status.as_deref() == Some("closed")), + ); + seen.add("archived neighbours", usize::from(archived)); + seen.add("labelled neighbours", usize::from(!n.labels.is_empty())); + seen.add( + "goal neighbours with an order", + usize::from(n.custom.contains_key("order")), + ); + seen.add( + "neighbours with a parent", + usize::from(n.item.belongs_to.is_some()), + ); + } + for e in &graph.edges { + let into_candidate = candidates.contains(e.to.as_str()); + let from_neighbour = graph.neighbours.contains_key(&e.from); + seen.add( + "inbound contributes_to to a candidate", + usize::from(into_candidate && e.kind == EdgeKind::ContributesTo), + ); + seen.add( + "mentions of a candidate from an inactive item", + usize::from(into_candidate && from_neighbour && e.kind == EdgeKind::Mentions), + ); + seen.add( + "inactive children of a candidate", + usize::from(into_candidate && from_neighbour && e.kind == EdgeKind::Parent), + ); + seen.add( + "dep edges from a candidate to an inactive blocker", + usize::from( + candidates.contains(e.from.as_str()) + && graph.neighbours.contains_key(&e.to) + && e.kind == EdgeKind::Dep, + ), + ); + } + seen.add( + "edges dropped for a deleted end", + count( + &s, + "SELECT COUNT(*) FROM present_edges e JOIN items i + ON i.id IN (e.from_id, e.to_id) WHERE i.deleted = '1'", + ), + ); + // A child with two present parent edges: only `present_parent`'s pick may count. + seen.add( + "children parented twice", + count( + &s, + "SELECT COUNT(*) FROM (SELECT from_id FROM present_edges WHERE kind = 'parent' + GROUP BY from_id HAVING COUNT(*) > 1)", + ), + ); + seen.add( + "removed edges", + count(&s, "SELECT COUNT(*) FROM edge_removes"), + ); + // A derived key decides: a project ranked ahead of one with a better priority. + let projects: Vec<&ItemRow> = want + .iter() + .filter(|i| i.item_type.as_deref() == Some("project")) + .collect(); + seen.add( + "projects ordered against their priority", + projects + .windows(2) + .filter(|w| w[0].priority > w[1].priority) + .count(), + ); + } + seen.assert_all(); + for what in [ + "closed neighbours", + "archived neighbours", + "inbound contributes_to to a candidate", + "mentions of a candidate from an inactive item", + "inactive children of a candidate", + "dep edges from a candidate to an inactive blocker", + "edges dropped for a deleted end", + "children parented twice", + "projects ordered against their priority", + ] { + assert!(seen.0.contains_key(what), "{what} is counted"); + } +} + +#[test] +fn reading_the_ranking_graph_costs_one_query_per_ticket_and_per_live_neighbour() { + // §9 "bounded cost": every edge touching an active ticket is read once, from its entries; + // each inactive neighbour costs one GetItem and, when live, one Query for its parent plus one + // per join asked for — and nothing grows with history: an edge added and removed many times + // is still one entry per add, read in the same Query. + let folder = Folder::platform().unwrap(); + let cfg = cfg(); + let reads = read::next_ranking_reads(&cfg).unwrap().unwrap(); + let mut s = Store::open_in_memory(1); + s.create_item("ab12.g000", "goal", "Closed goal", "u"); + s.set_field("ab12.g000", "status", Some("closed".into()), "u"); + s.create_item("ab12.g001", "goal", "Deleted goal", "u"); + let n = 10u64; + for i in 0..n { + let id = format!("ab12.p{i:03}"); + s.create_item(&id, "project", "P", "u"); + for round in 0..5 { + s.add_edge(&id, "ab12.g000", EdgeKind::ContributesTo, "u"); + if round < 4 { + s.remove_edge(&id, "ab12.g000", EdgeKind::ContributesTo, "u"); + } + } + s.add_edge(&id, "ab12.g001", EdgeKind::Mentions, "u"); + } + s.delete_item("ab12.g001", "u"); + let t = fold(&folder, &delivered(&s, 1)); + let plain = ready(records::active_board(&t, ready(board::select(&t)).unwrap())).unwrap(); + let before = t.requests(); + let server = ready(records::with_ranking(&t, plain, &reads)).unwrap(); + let after = t.requests(); + // One adjacency Query per active ticket; the live neighbour's parent, labels and custom values. + assert_eq!(after.queries - before.queries, n + 3); + // One row per inactive neighbour: the closed goal and the deleted one. + assert_eq!(after.gets - before.gets, 2); + assert_eq!(after.index_queries, before.index_queries); + assert_eq!(after.writes, before.writes); + let local = read::with_ranking(&s, read::active_board(&s).unwrap(), &reads).unwrap(); + assert_eq!(server.ranking, local.ranking); + let graph = server.ranking.unwrap(); + assert_eq!( + graph.edges.len(), + n as usize, + "one contributes_to per project" + ); + assert_eq!( + graph.neighbours.keys().collect::>(), + ["ab12.g000"], + "the closed goal is a neighbour, the deleted one is not" + ); +} + +#[test] +fn a_stream_folded_by_index_revision_2_is_refused_then_refolded_and_ranks() { + // A stream folded before the adjacency entries said what their edge is (index revision 2): its + // entries hold only the tag, which would read no edge at all — so it is refused, and the fold + // run's rule refolds it. + let folder = Folder::platform().unwrap(); + let cfg = cfg(); + let reads = read::next_ranking_reads(&cfg).unwrap().unwrap(); + for seed in [3, 11, 27] { + let s = gen_board(seed); + let ops = delivered(&s, seed); + let t = fold(&folder, &ops); + for (sk, row) in t.rows() { + if let Some(Cell::Text(tag)) = row.get("tag").filter(|_| sk.starts_with(".adj#")) { + ready( + t.write(&Write { + sk: sk.clone(), + set: vec![("tag".into(), Cell::Text(tag.clone()))], + remove: ["from_id", "to_id", "kind", "lamport", "site", "removed"] + .map(String::from) + .to_vec(), + condition: Some(Cond::RowPresent), + }), + ) + .unwrap(); + } + } + ready(t.write(&Write { + sk: WATERMARK_KEY.into(), + set: vec![(format!("revision_{INDEX_DOMAIN}"), Cell::Int(2))], + remove: Vec::new(), + condition: Some(Cond::RowPresent), + })) + .unwrap(); + + let selection = ready(board::select(&t)).unwrap(); + match ready(records::active_board(&t, selection)) { + Err(records::RecordsError::NotCurrent { found: Some(2) }) => {} + other => panic!("seed {seed}: a revision-2 stream must be refused, got {other:?}"), + } + let mark = ready(Folder::watermark(&t)).unwrap().unwrap(); + assert!(!folder.is_current(&mark), "seed {seed}"); + ready(Folder::clear(&t)).unwrap(); + ready(folder.fold_batch(&t, &ops)).unwrap(); + + let server = served(&t, &reads); + assert_eq!( + ids(&read::next_active(&cfg, &server, NOW).unwrap()), + ids(&read::next(&cfg, &s, NOW, None).unwrap()), + "seed {seed}" + ); + } +} diff --git a/docs/specs/derived-ranking-keys.md b/docs/specs/derived-ranking-keys.md index 22c2d7f..cf37db7 100644 --- a/docs/specs/derived-ranking-keys.md +++ b/docs/specs/derived-ranking-keys.md @@ -2,7 +2,9 @@ **Status: decided by the owner on 2026-10-09** (§10). The cuts that build it are listed in §11. **Built locally (x0wf, §12):** the three key kinds, their load-time rules and the type chain over -the SQL path, in `next` and `list`. The server cuts (1c5s, xkr7) are open. +the SQL path, in `next` and `list`. **Built on the server for `next` (1c5s, §6):** the same keys +over the hosted board, with the rule the owner decided on 2026-10-11 (§10.5). The `typed` index and +a ranked hosted `list` (xkr7) are open. Fourth cut of `6j6v.kfs7` (cut with the owner on 2026-10-09). The three cuts before it add the plumbing this one ranks with: @@ -135,11 +137,41 @@ derived key pays nothing. `nxs-fold-ddb` answers `next` from a sparse index of ACTIVE tickets through `graph::Board` (6j6v.k7w7). Derived keys must give the same order there. -**`next`.** -- `Board` gains the edge kinds the keys need (`contributes_to`, `mentions`), and each ticket's - type, status, labels and the fields the keys read. -- The rule that makes this possible is the one `Board` already has: a counterpart that is not an - active ticket counts as absent, which is exactly §4's scope for `next`. +**`next`** (built in 1c5s; decided by the owner on 2026-10-11, §10.5). +- **A key reads every LIVE neighbour, on the server as locally** (§12, "Live neighbours"): a + deleted neighbour is gone, a closed or archived one counts. This replaces the first reading of + this section, that a counterpart which is not an active ticket counts as absent. That rule stays + the rule of the LANES (`graph::Board`); it is not the rule of the keys. Only the scope of + `of = "rank"` is the active set: a neighbour outside `next`'s candidates has no rank (§4), on + both paths. +- **Where it lives.** The facade's `ActiveBoard` — the store-free lanes' input, `graph::Board` + plus each active ticket's row and joins — gains a `RankingGraph`: every present edge of the kinds + the keys name (`contributes_to`, `mentions`, `dep`, `parent` as the `present_parent` projection) + with an active end and both ends live, and each live neighbour that is not active with its row, + labels and custom values. `graph::Board` itself is unchanged: it holds no rows, and the lanes read + no neighbour that is not active. `read::next_ranking_reads(cfg)` names what the graph must hold + for a plugin; `read::next_active` refuses a plugin with derived keys while its board lacks it, and + ranks by the same evaluation the store path runs once it is there. +- **What the fold writes.** Every edge add already had an adjacency entry under each end + (`.adj##`), whatever its kind and its ends' activity. Since index revision 3 each entry + also copies the add's `from_id`, `to_id`, `kind`, `lamport` and `site` and says whether the add is + removed, kept up to date when the remove is folded. So one Query on an item's prefix reads every + edge touching it, in and out, with its presence: inbound `contributes_to`, `mentions` both ways, + and the `parent` edges a child's current parent is picked from. No new index, no new key + attribute: the TABLE CONTRACT IS UNCHANGED. Revision 3 makes every stream folded before it not + current; it answers `NotCurrent` until the fold run has folded it again. +- **What the server reads** (`records::with_ranking`): one Query per active ticket on its + adjacency; per neighbour that is not active one `GetItem` of its row and, when it is live, one + Query on its own adjacency for its current parent, plus one Query each for its labels and custom + values when a key reads them. +- **Refused on the server until xkr7:** a `[ranking.next]` order that reads — directly or down its + type chain — the rank of a type only `[ranking.list]` orders (§12, first point). That rank is + computed over the type's live items, which the active board does not hold. The store path ranks + such a plugin. Every other derived `next` key ranks on the server. +- **What xkr7 builds on.** The self-describing adjacency entries are written for EVERY edge between + ANY items, so xkr7's "widened adjacency" needs no new entries for the edges: a `list`-ranked item + reads its edges with the same Query. What xkr7 adds is the `typed` index (the `list` scope of a + type), the read that ranks a type's live items from it, and lifting the refusal above. **`list` needs a second index (decided §10.4).** Objects that `list` ranks, such as a person, are long-lived and have no "active" status. If they did, they would rank through `next`, and `list` @@ -155,10 +187,12 @@ gains a second sparse index, keyed by type: - **Generic.** The fold indexes every live item by its type, so it needs no plugin knowledge of which types `list` ranks. The price is one index write per item, not only for the ranked types. - **Edges.** A derived key on a `list`-ranked type reads that item's `contributes_to` and `mentions` - neighbours, and the counts read their inbound edges. Today the adjacency entries (`.adj#`) are - kept only for `dep`/`parent` edges that touch an active ticket. They are widened to every present - edge of a kind some key names, between live items. That widening is the open design work of the - server cut (§11), together with what it costs per edge write. + neighbours, and the counts read their inbound edges. Corrected in 1c5s: the adjacency entries + (`.adj#`) were always written for every edge add, under both ends; only the `active` INDEX flag is + limited to `dep`/`parent` edges with an active end. Since index revision 3 the entries say what + their edge is and whether it is removed (`next` above), so a `list`-ranked item reads its edges + with one Query on its own prefix, at no extra cost per edge write. What stays open for xkr7 is + the read of a type's live items through `typed` and their neighbours' rows. - **Archived items.** They are in the index, because they are live. `list` filters them at read, as it does locally. - **It changes the table contract.** The deploying stack (the web package's CDK) must add the GSI @@ -222,8 +256,10 @@ the plugin declares (`plugin-custom-fields.md`), as decided in §10.2. - **Per kind:** unit tests over hand-built boards: a value from one neighbour, from several (`min`/`max`), from none (null placement); counts with each filter. - **Chains:** actions by project rank by goal order across three types, and a refused cycle. -- **Differential:** SQL path against `graph::Board` on generated boards with derived keys, as - `fold_differential` does for the lanes. +- **Differential:** SQL path against the server path on generated boards with derived keys, as + `fold_differential` does for the lanes. Built for `next` in 1c5s + (`crates/fold-ddb/tests/derived_differential.rs`): the SQL path, the store-free path over the + store's own board and graph, and the same over the folded tables give one order. - **Determinism:** shuffled edge order gives the same result, and ties fall to the id. - **Bounded cost:** a ratio guard in the style of `xn8s`. A query with derived keys runs the edge pass once, not once per candidate. @@ -245,13 +281,24 @@ the plugin declares (`plugin-custom-fields.md`), as decided in §10.2. items was declined: the objects `list` ranks have no active status by design. If they had one, they would rank through `next`, and `list` would not be needed. Leaving `list` unranked on the server was declined because the product reads through the hosted path. +5. **On the server a key reads every live neighbour** (owner, 2026-10-11, during 1c5s). The first + text of §6 said a counterpart that is not an active ticket counts as absent; the local build + (§12) reads every live neighbour, closed and archived ones included. The owner chose the local + rule for both paths, so a server and a replica rank alike: the fold keeps self-describing + adjacency entries (index revision 3) and the server reads the neighbours that are not active. + Narrowing the local rule to active neighbours was declined, as was refusing on the server every + key whose value can depend on a neighbour that is not active (it would refuse §8's first order). + A `[ranking.next]` order that reads the rank of a type only `[ranking.list]` orders is refused on + the server until the `typed` index (xkr7) gives it that type's scope. ## 11. Cuts - **Derived keys, local.** The three key kinds, load-time validation and the type chain over the SQL path. Usable in `next` (after `z9jk`) and `list` (after `n698`). -- **Derived keys on the server, `next`.** `graph::Board` with the extra edges and fields; the - differential test. +- **Derived keys on the server, `next`** (1c5s, built). The `RankingGraph` on the store-free + lanes' board, the self-describing adjacency entries (index revision 3), and the differential test + (SQL path = Board path = fold-ddb path on generated boards with closed, archived and deleted + neighbours, mentions and inbound `contributes_to`). - **The `typed` index and a ranked hosted `list`.** The GSI, the widened adjacency, the read, and the table-contract note for the deploying stack, coordinated with app-foundations. - **Shared with `6j6v.q29p`.** The reverse mention lookup and `count = "mentioned_by"` use one @@ -286,7 +333,8 @@ reading of the text, so a later decision can loosen it without breaking a declar `blocked` and the lanes' `--sort rank`, which compute none. The `[ranking.list]` default order takes them. - **`nulls` on a `count` or `sum`** is refused: a count is never null (§3.2). -- **The store-free lanes** (`read::next_active`) refuse a `[ranking.next]` with derived keys until - the server cut (1c5s) gives them the edges. +- **The store-free lanes** (`read::next_active`) rank a `[ranking.next]` with derived keys from + the board's `RankingGraph` since 1c5s (§6), with the live-neighbour rule above; they refuse only + an order that reads the rank of a type only `[ranking.list]` orders, until xkr7. - **TOML.** An inline table may not span lines outside its arrays (TOML 1.0), so a `sum`'s `dir` stands on the line its array closes on. §8 above breaks that line for reading only. From 9fb48191d0dd9900f0cd79d46bdd863a46f96902 Mon Sep 17 00:00:00 2001 From: Carsten Koch Date: Sun, 11 Oct 2026 20:04:29 +0200 Subject: [PATCH 2/2] The typed index and a ranked hosted list (nxf 6j6v.xkr7) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit A hosted board answers `list` ranked by `[ranking.list]`, derived keys included, as a replica does, and a hosted `next` whose keys read a `[ranking.list]` rank no longer is refused (spec §6, decision §10.4). - fold-ddb: third sparse GSI `typed` (nxf_typed = #, the type escaped; sort key sk; projection ALL) on every live item row, closed and archived included, kept by refresh_item after the row. Table::query_typed (MemTable, DynamoDbTable, create_table). A type whose partition passes 2,048 bytes is refused at fold time. Ships in index revision 3 with the self-describing .adj# entries (#54, unreleased): no revision 4. records::list_board reads a ListBoard; the graph builder is shared with with_ranking and reads the live items of RankingReads::list_types off the index. - facade: ListReads, ListItem, ListBoard, list_reads, list_board (store oracle), list_board_page, list_board_value; RankingReads::list_types, RankingGraph::listed (additive); next_ranking_reads no longer refuses a next order reading a list-only type's rank. Not paged, as the hosted next: limit cuts, paginate/token are validation. - No new adjacency: the revision-3 .adj# entries carry everything. - tests: list_differential (SQL = Board = fold-ddb for list and next on generated boards with retypes, cross-section ranks, JSON, cost guard, revision refusal), typed index in fold_differential, robustness and dynamodb_local, facade seam tests. - docs: crate table contract, E4 §5.1, spec §6/§9/§11/§12; 1c5s fragment aligned (contract change and lifted refusal). Co-Authored-By: Claude Opus 5.5 Claude-Session: https://claude.ai/code/session_016q5uMUkZJQW4fckJoA3FME --- changes/1c5s-server-derived-keys.md | 4 +- changes/xkr7-typed-index.md | 8 + crates/facade/src/read.rs | 380 ++++++++++++++++-- crates/facade/src/read/derived.rs | 402 ++++++++++++++----- crates/facade/tests/derived_ranking.rs | 196 ++++++++- crates/fold-ddb/src/board.rs | 49 ++- crates/fold-ddb/src/ddb.rs | 27 +- crates/fold-ddb/src/fold.rs | 23 +- crates/fold-ddb/src/layout.rs | 17 +- crates/fold-ddb/src/lib.rs | 36 +- crates/fold-ddb/src/mem.rs | 16 +- crates/fold-ddb/src/records.rs | 409 ++++++++++++++++--- crates/fold-ddb/src/table.rs | 11 +- crates/fold-ddb/tests/common/mod.rs | 11 +- crates/fold-ddb/tests/dynamodb_local.rs | 20 +- crates/fold-ddb/tests/fold_differential.rs | 28 +- crates/fold-ddb/tests/list_differential.rs | 441 +++++++++++++++++++++ crates/fold-ddb/tests/robustness.rs | 108 ++++- docs/specs/E4-sync-protocol-server.md | 7 + docs/specs/derived-ranking-keys.md | 74 +++- 20 files changed, 2020 insertions(+), 247 deletions(-) create mode 100644 changes/xkr7-typed-index.md create mode 100644 crates/fold-ddb/tests/list_differential.rs diff --git a/changes/1c5s-server-derived-keys.md b/changes/1c5s-server-derived-keys.md index 1d63ec8..58aed9e 100644 --- a/changes/1c5s-server-derived-keys.md +++ b/changes/1c5s-server-derived-keys.md @@ -3,6 +3,6 @@ type: added facade: changed --- [en] -A hosted board now ranks `next` by derived ranking keys (`related`, `count`, `sum`) exactly as a local replica does. The store-free `next` (`read::next_active`, `next_active_value`, `next_active_filtered`) ranks a plugin with such keys from a new `ActiveBoard::ranking` graph: the edges around the active tickets and the live neighbours they reach, closed and archived ones included, deleted ones not. A server fills it with `nxs_fold_ddb::records::with_ranking` for what `read::next_ranking_reads(cfg)` names; `read::with_ranking` reads the same graph from a store. Without the graph, `next_active` still refuses such a plugin. One case stays refused on the store-free path: a `[ranking.next]` order that reads the rank of a type only `[ranking.list]` orders — answer that `next` from a store. Everything is additive: `ActiveBoard` gains a field (it is `#[non_exhaustive]`), and `RankingReads`, `RankingGraph`, `Neighbour`, `next_ranking_reads` and `with_ranking` are new. For a server's operators: the fold's index revision goes from 2 to 3, so every hosted stream is not current and is folded again on its next run; until then its records read answers `NotCurrent`. The table contract is unchanged: no new index and no new key attribute, only more attributes on the existing `.adj#` adjacency entries. +A hosted board now ranks `next` by derived ranking keys (`related`, `count`, `sum`) exactly as a local replica does. The store-free `next` (`read::next_active`, `next_active_value`, `next_active_filtered`) ranks a plugin with such keys from a new `ActiveBoard::ranking` graph: the edges around the active tickets and the live neighbours they reach, closed and archived ones included, deleted ones not. A server fills it with `nxs_fold_ddb::records::with_ranking` for what `read::next_ranking_reads(cfg)` names; `read::with_ranking` reads the same graph from a store. Without the graph, `next_active` still refuses such a plugin. Everything is additive: `ActiveBoard` gains a field (it is `#[non_exhaustive]`), and `RankingReads`, `RankingGraph`, `Neighbour`, `next_ranking_reads` and `with_ranking` are new. For a server's operators: the fold's index revision goes from 2 to 3, so every hosted stream is not current and is folded again on its next run; until then its records read answers `NotCurrent`. The `.adj#` adjacency entries gain attributes; the one table-contract change of this release, the `typed` index, is described with the hosted `list`. [de] -Ein gehostetes Brett rankt `next` jetzt nach abgeleiteten Ranking-Schlüsseln (`related`, `count`, `sum`) genau so wie ein lokales Replikat. Das speicherfreie `next` (`read::next_active`, `next_active_value`, `next_active_filtered`) rankt ein Plugin mit solchen Schlüsseln aus einem neuen Graphen `ActiveBoard::ranking`: den Kanten um die aktiven Tickets und den lebenden Nachbarn, die sie erreichen, geschlossene und archivierte eingeschlossen, gelöschte nicht. Ein Server füllt ihn mit `nxs_fold_ddb::records::with_ranking` für das, was `read::next_ranking_reads(cfg)` nennt; `read::with_ranking` liest denselben Graphen aus einem Speicher. Ohne den Graphen lehnt `next_active` ein solches Plugin weiterhin ab. Ein Fall bleibt auf dem speicherfreien Weg abgelehnt: eine Reihenfolge in `[ranking.next]`, die den Rang eines Typs liest, den nur `[ranking.list]` ordnet — dieses `next` beantwortet ein Speicher. Alles ist additiv: `ActiveBoard` bekommt ein Feld (es ist `#[non_exhaustive]`), und `RankingReads`, `RankingGraph`, `Neighbour`, `next_ranking_reads` und `with_ranking` sind neu. Für den Betrieb eines Servers: Die Index-Revision des Folds steigt von 2 auf 3, daher ist jeder gehostete Stream nicht aktuell und wird bei seinem nächsten Lauf neu gefaltet; bis dahin antwortet sein Records-Lesen mit `NotCurrent`. Der Tabellenvertrag bleibt unverändert: kein neuer Index und kein neues Schlüsselattribut, nur mehr Attribute an den bestehenden `.adj#`-Adjazenzeinträgen. +Ein gehostetes Brett rankt `next` jetzt nach abgeleiteten Ranking-Schlüsseln (`related`, `count`, `sum`) genau so wie ein lokales Replikat. Das speicherfreie `next` (`read::next_active`, `next_active_value`, `next_active_filtered`) rankt ein Plugin mit solchen Schlüsseln aus einem neuen Graphen `ActiveBoard::ranking`: den Kanten um die aktiven Tickets und den lebenden Nachbarn, die sie erreichen, geschlossene und archivierte eingeschlossen, gelöschte nicht. Ein Server füllt ihn mit `nxs_fold_ddb::records::with_ranking` für das, was `read::next_ranking_reads(cfg)` nennt; `read::with_ranking` liest denselben Graphen aus einem Speicher. Ohne den Graphen lehnt `next_active` ein solches Plugin weiterhin ab. Alles ist additiv: `ActiveBoard` bekommt ein Feld (es ist `#[non_exhaustive]`), und `RankingReads`, `RankingGraph`, `Neighbour`, `next_ranking_reads` und `with_ranking` sind neu. Für den Betrieb eines Servers: Die Index-Revision des Folds steigt von 2 auf 3, daher ist jeder gehostete Stream nicht aktuell und wird bei seinem nächsten Lauf neu gefaltet; bis dahin antwortet sein Records-Lesen mit `NotCurrent`. Die `.adj#`-Adjazenzeinträge bekommen Attribute; die eine Änderung des Tabellenvertrags in diesem Release, der Index `typed`, ist bei der gehosteten `list` beschrieben. diff --git a/changes/xkr7-typed-index.md b/changes/xkr7-typed-index.md new file mode 100644 index 0000000..31fb4a1 --- /dev/null +++ b/changes/xkr7-typed-index.md @@ -0,0 +1,8 @@ +--- +type: added +facade: changed +--- +[en] +A hosted board now answers `list` ranked by the plugin's `[ranking.list]`, derived ranking keys included, exactly as a local replica does. The facade names what to read for a query (`read::list_reads`, a new `ListReads`), a server reads it into a new `ListBoard` with `nxs_fold_ddb::records::list_board`, and `read::list_board_page` / `read::list_board_value` filter (type, container, label, status), rank and project it with the code `list` runs locally; `read::list_board` reads the same board from a store. The hosted list is not paged: `limit` cuts it with the total disclosed, and `paginate` or a token is refused, as for the hosted `next`. A hosted `next` whose keys read the rank `[ranking.list]` gives a type now ranks too instead of being refused: `RankingReads` gains `list_types` and `RankingGraph` gains `listed` (both `#[non_exhaustive]`, so additive). For a server's operators, the table contract changes: a third sparse index, GSI `typed` — partition key `nxf_typed` (string, `#`), sort key `sk` (string), projection ALL — set by the fold on every live item row (closed and archived included, removed on delete). Add the index to the stack first, then deploy the read; a fold against a table without the index writes the attribute harmlessly. It ships in fold index revision 3 together with the self-describing adjacency entries, so one refold covers both. +[de] +Ein gehostetes Brett beantwortet `list` jetzt nach dem `[ranking.list]` des Plugins gerankt, abgeleitete Ranking-Schlüssel eingeschlossen, genau so wie ein lokales Replikat. Die Fassade nennt, was für eine Anfrage zu lesen ist (`read::list_reads`, ein neues `ListReads`), ein Server liest es mit `nxs_fold_ddb::records::list_board` in ein neues `ListBoard`, und `read::list_board_page` / `read::list_board_value` filtern (Typ, Container, Label, Status), ranken und projizieren es mit dem Code, den `list` lokal ausführt; `read::list_board` liest dasselbe Brett aus einem Speicher. Die gehostete Liste wird nicht geblättert: `limit` schneidet sie mit offengelegter Gesamtzahl, `paginate` oder ein Token wird abgelehnt, wie beim gehosteten `next`. Ein gehostetes `next`, dessen Schlüssel den Rang lesen, den `[ranking.list]` einem Typ gibt, rankt jetzt ebenfalls, statt abgelehnt zu werden: `RankingReads` bekommt `list_types` und `RankingGraph` bekommt `listed` (beide `#[non_exhaustive]`, also additiv). Für den Betrieb eines Servers ändert sich der Tabellenvertrag: ein dritter dünn besetzter Index, GSI `typed` — Partitionsschlüssel `nxf_typed` (String, `#`), Sortierschlüssel `sk` (String), Projektion ALL —, den der Fold an jeder lebenden Item-Zeile setzt (geschlossene und archivierte eingeschlossen, beim Löschen entfernt). Zuerst den Index in den Stack aufnehmen, dann das Lesen ausrollen; ein Fold gegen eine Tabelle ohne den Index schreibt das Attribut folgenlos. Er kommt mit der Fold-Index-Revision 3 zusammen mit den selbstbeschreibenden Adjazenzeinträgen, ein Neufalten deckt also beides ab. diff --git a/crates/facade/src/read.rs b/crates/facade/src/read.rs index 7784967..6f3ec09 100644 --- a/crates/facade/src/read.rs +++ b/crates/facade/src/read.rs @@ -2102,6 +2102,20 @@ fn blocked_record(cfg: Option<&PluginConfig>, row: &BlockedItem, j: &RecordJoins rec } +/// One `list` record without a store (6j6v.xkr7): what [`list_to_value_with_custom`] gives — the +/// canonical item, the sparse `labels` and `parent_closed_reason` joins, and the lane joins. +fn list_record(cfg: Option<&PluginConfig>, item: &ItemRow, j: &RecordJoins) -> Value { + let mut rec = item_value(item); + if !j.labels.is_empty() { + rec["labels"] = json!(j.labels); + } + if let Some(reasons) = parent_closed_reason_value(j.parent_closed) { + rec["parent_closed_reason"] = reasons; + } + put_lane_joins(cfg, &mut rec, j); + rec +} + /// One record of a bare lane (`deferred`/`closed`/`archived`/`search`): the canonical item and the /// lane joins. fn lane_record(cfg: Option<&PluginConfig>, item: &ItemRow, j: &RecordJoins) -> Value { @@ -2800,6 +2814,10 @@ pub struct RankingReads { pub labels: bool, /// An `of` reads a custom field: the neighbours' custom values. pub custom: bool, + /// The types whose `[ranking.list]` rank a key reads (6j6v.xkr7): the graph also holds every + /// live, unarchived item of each — the type's `list` scope ([`RankingGraph::listed`]) — and the + /// edges around them. A server reads them from its `typed` index. + pub list_types: BTreeSet, } impl RankingReads { @@ -2808,21 +2826,29 @@ impl RankingReads { needed.kinds.is_subset(&self.kinds) && (self.labels || !needed.labels) && (self.custom || !needed.custom) + && needed.list_types.is_subset(&self.list_types) } } -/// The graph around a board's active tickets that the derived ranking keys read (6j6v.1c5s, +/// The graph around a board's rows that the derived ranking keys read (6j6v.1c5s, /// docs/specs/derived-ranking-keys.md §6): the input [`next_active`] ranks a `[ranking.next]` with -/// `related` / `count` / `sum` keys from, so a server ranks exactly as [`next`] does over a store. +/// `related` / `count` / `sum` keys from, and [`list_board_page`] a `[ranking.list]` (6j6v.xkr7), +/// so a server ranks exactly as [`next`] and [`list_page`] do over a store. +/// +/// The SUBJECTS are the rows a key is evaluated on: the board's own rows — an [`ActiveBoard`]'s +/// active tickets; a [`ListBoard`]'s listed items and, when it carries one, its active board's +/// tickets — and the `listed` items below. /// -/// - `edges`: every present edge of a kind in `reads.kinds` with at least one ACTIVE end (an -/// [`ActiveBoard::tickets`] entry) and both ends LIVE (not deleted, not missing). A `parent` edge -/// is the `present_parent` projection — a child's one current parent, the one `belongs_to` reads — -/// so a child re-parented concurrently is under one parent only. Distinct, ordered. -/// - `neighbours`: the other ends of `edges` that are not active tickets — closed and archived -/// items included, since a key reads every LIVE neighbour (owner, 2026-10-11) — each with its row -/// (`belongs_to` filled as the store fills it), its labels when `reads.labels` and its non-empty -/// custom values when `reads.custom`, else empty. +/// - `edges`: every present edge of a kind in `reads.kinds` with at least one end a subject and +/// both ends LIVE (not deleted, not missing). A `parent` edge is the `present_parent` projection — +/// a child's one current parent, the one `belongs_to` reads — so a child re-parented concurrently +/// is under one parent only. Distinct, ordered. +/// - `listed`: the ids of every live, unarchived item of a type in `reads.list_types` — the `list` +/// scope a `[ranking.list]` rank is computed over (6j6v.xkr7). Empty without list types. +/// - `neighbours`: every live row the board does not hold itself — the other ends of `edges` and +/// the `listed` items, closed and archived items included, since a key reads every LIVE neighbour +/// (owner, 2026-10-11) — each with its row (`belongs_to` filled as the store fills it), its +/// labels when `reads.labels` and its non-empty custom values when `reads.custom`, else empty. /// /// `#[non_exhaustive]`: build it with [`RankingGraph::new`] and set the fields. #[derive(Debug, Clone, Default, PartialEq, Eq)] @@ -2833,6 +2859,8 @@ pub struct RankingGraph { pub edges: BTreeSet, /// By id. pub neighbours: BTreeMap, + /// The live, unarchived items of `reads.list_types` (6j6v.xkr7), by id. + pub listed: BTreeSet, } impl RankingGraph { @@ -2845,8 +2873,8 @@ impl RankingGraph { } } -/// A live item next to an active ticket that is not itself active: what a derived ranking key reads -/// of it ([`RankingGraph::neighbours`]). +/// A live item a derived ranking key reads that the board does not hold itself — next to one of +/// its rows, or in the `list` scope of a type ([`RankingGraph::neighbours`]). /// /// `#[non_exhaustive]`: build it with [`Neighbour::new`] and set the fields. #[derive(Debug, Clone, PartialEq, Eq)] @@ -2874,10 +2902,10 @@ impl Neighbour { /// [`RankingReads`] a reader fills [`ActiveBoard::ranking`] for — or `None` when `[ranking.next]` /// declares no derived ranking key (6j6v.1c5s). It does not depend on the board. /// -/// `validation` for a `[ranking.next]` that reads, directly or down its type chain, the rank of a -/// type only `[ranking.list]` orders (docs/specs/derived-ranking-keys.md §6): that rank is computed -/// over the type's live items, and a board of active tickets does not hold them. The store path -/// ([`next`]) ranks such a plugin. +/// A `[ranking.next]` that reads, directly or down its type chain, the rank of a type only +/// `[ranking.list]` orders names that type in [`RankingReads::list_types`] (6j6v.xkr7): that rank +/// is computed over the type's live items, which the reader puts in the graph. Until 6j6v.xkr7 +/// such a plugin was refused here; it is no longer, and the `Result` stays for a later refusal. pub fn next_ranking_reads(cfg: &PluginConfig) -> Result> { derived::reads_for_next(cfg) } @@ -2885,23 +2913,49 @@ pub fn next_ranking_reads(cfg: &PluginConfig) -> Result> { /// `board` with its [`RankingGraph`] for `reads` read from `store` (6j6v.1c5s) — the parity oracle /// for `nxs_fold_ddb::records::with_ranking`, as [`active_board`] is for the records. One edge pass /// ([`Store::present_edges_of_kinds`]), one bulk read of the neighbours' rows, and one of their -/// labels and of their custom values when `reads` asks for them. +/// labels and of their custom values when `reads` asks for them; with list types, one read of the +/// live items (6j6v.xkr7). pub fn with_ranking( store: &Store, mut board: ActiveBoard, reads: &RankingReads, ) -> Result { + let held: BTreeSet<&str> = board.tickets.keys().map(String::as_str).collect(); + board.ranking = Some(ranking_graph(store, reads, &held)?); + Ok(board) +} + +/// The [`RankingGraph`] for `reads` around the rows a board holds itself (`held`, every one a +/// subject), read from `store` — what [`with_ranking`] and [`list_board`] put on their boards. +fn ranking_graph( + store: &Store, + reads: &RankingReads, + held: &BTreeSet<&str>, +) -> Result { + let listed: Vec = if reads.list_types.is_empty() { + Vec::new() + } else { + derived::list_scope(store)? + .into_iter() + .filter(|i| { + i.item_type + .as_deref() + .is_some_and(|t| reads.list_types.contains(t)) + }) + .collect() + }; + let listed_ids: BTreeSet<&str> = listed.iter().map(|i| i.id.as_str()).collect(); + let subject = |id: &str| held.contains(id) || listed_ids.contains(id); let kinds: Vec = reads.kinds.iter().copied().collect(); - let active = |id: &str| board.tickets.contains_key(id); let touching: Vec<(String, String, EdgeKind)> = store .present_edges_of_kinds(&kinds)? .into_iter() - .filter(|(from, to, _)| active(from) || active(to)) + .filter(|(from, to, _)| subject(from) || subject(to)) .collect(); let others: BTreeSet<&str> = touching .iter() .flat_map(|(from, to, _)| [from.as_str(), to.as_str()]) - .filter(|id| !active(id)) + .filter(|id| !subject(id)) .collect(); let others: Vec<&str> = others.into_iter().collect(); let live: BTreeMap = store @@ -2913,11 +2967,17 @@ pub fn with_ranking( let edges: BTreeSet = touching .into_iter() .filter(|(from, to, _)| { - (active(from) || live.contains_key(from)) && (active(to) || live.contains_key(to)) + (subject(from) || live.contains_key(from)) && (subject(to) || live.contains_key(to)) }) .map(|(from, to, kind)| nexus_flow_core::graph::Edge { from, to, kind }) .collect(); - let ids: Vec<&str> = live.keys().map(String::as_str).collect(); + let mut rows = live; + for item in &listed { + if !held.contains(item.id.as_str()) { + rows.insert(item.id.clone(), item.clone()); + } + } + let ids: Vec<&str> = rows.keys().map(String::as_str).collect(); let mut labels = if reads.labels { store.labels_of_bulk(&ids)? } else { @@ -2930,7 +2990,8 @@ pub fn with_ranking( }; let mut graph = RankingGraph::new(reads.clone()); graph.edges = edges; - graph.neighbours = live + graph.listed = listed_ids.iter().map(|id| id.to_string()).collect(); + graph.neighbours = rows .into_iter() .map(|(id, item)| { let mut n = Neighbour::new(item); @@ -2939,8 +3000,7 @@ pub fn with_ranking( (id, n) }) .collect(); - board.ranking = Some(graph); - Ok(board) + Ok(graph) } /// The store path's input as an [`ActiveBoard`]: the store's selection ([`graph::select`]) and, for @@ -2987,10 +3047,11 @@ pub fn active_board(store: &Store) -> Result { /// [`Engine::next`]: crate::engine::Engine::next /// /// A `[ranking.next]` with derived ranking keys (6j6v.x0wf) is ranked from the board's -/// [`RankingGraph`] (6j6v.1c5s), by the same evaluation the store path runs. It is refused with -/// `validation` when the board carries no graph, or one read for less than -/// [`next_ranking_reads`] names — ranking without those edges would order apart from the store path -/// in silence — and when [`next_ranking_reads`] refuses the plugin. +/// [`RankingGraph`] (6j6v.1c5s), by the same evaluation the store path runs — a key that reads the +/// rank of a type only `[ranking.list]` orders too, from the graph's listed items (6j6v.xkr7). It +/// is refused with `validation` when the board carries no graph, or one read for less than +/// [`next_ranking_reads`] names: ranking without those edges would order apart from the store path +/// in silence. pub fn next_active(cfg: &PluginConfig, board: &ActiveBoard, now: &str) -> Result> { crate::validate::iso_date(now)?; let graph = match next_ranking_reads(cfg)? { @@ -3115,6 +3176,267 @@ pub fn deferred_active_value(cfg: &PluginConfig, board: &ActiveBoard, now: &str) Ok(board.records(&items, |i, j| lane_record(Some(cfg), i, j))) } +// ---- list without a store (6j6v.xkr7) ---------------------------------------- +// +// The hosted `list`: the same rows in the same order as [`list_page`] over a store holding the same +// board, from a [`ListBoard`] a server reads off its `typed` index (docs/specs/derived-ranking-keys.md +// §6) — the listed types' live items with their joins, `next`'s board when a key reads a rank +// `next` gives, and the [`RankingGraph`] the derived keys read. As for the store-free `next`, the +// facade decides what to read ([`list_reads`]) and ranks and projects it; the server only reads. + +/// What the store-free `list` a [`ListQuery`] asks for reads (6j6v.xkr7) — the input of a reader +/// that builds the [`ListBoard`] ([`list_board`] from a store, `nxs_fold_ddb::records::list_board` +/// from the folded tables). [`list_reads`] gives it. +/// +/// `#[non_exhaustive]`: start from `ListReads::default()` and set the fields. +#[derive(Debug, Clone, Default, PartialEq, Eq)] +#[non_exhaustive] +pub struct ListReads { + /// The listed types: the board holds every live, unarchived item of each. `None`: every live, + /// unarchived item, whatever its type — a query without a type filter. + pub types: Option>, + /// A key reads the rank of a type `next` ranks: the board carries `next`'s board, the active + /// tickets and their [`Board`], to take `next`'s candidates from at the query's `now`. + pub next: bool, + /// The graph the derived keys read (`None`: none in play). Its subjects are the listed items, + /// the active tickets when `next` is set, and its own listed items + /// ([`RankingReads::list_types`]). + pub ranking: Option, +} + +/// One listed item and the joins its `list` record and the filters read (6j6v.xkr7) — the +/// per-item input of the store-free `list`. Every field holds what the store path reads for the +/// same item: `item` with its `belongs_to` (`present_parent`), `labels` as `present_labels`, +/// `custom` every non-empty custom value, the timestamps with an unstamped `''` as `None`, +/// `contributes_to` the present targets sorted and distinct, and `parent_closed` what +/// [`parent_closed_reason`] gives. +/// +/// `#[non_exhaustive]`: build it with [`ListItem::new`] and set the fields. +#[derive(Debug, Clone, PartialEq, Eq)] +#[non_exhaustive] +pub struct ListItem { + pub item: ItemRow, + pub labels: Vec, + pub custom: BTreeMap, + pub created_at: Option, + pub updated_at: Option, + pub contributes_to: Vec, + pub parent_closed: Vec<(String, Option)>, +} + +impl ListItem { + /// An item with no joins yet. + pub fn new(item: ItemRow) -> ListItem { + ListItem { + item, + labels: Vec::new(), + custom: BTreeMap::new(), + created_at: None, + updated_at: None, + contributes_to: Vec::new(), + parent_closed: Vec::new(), + } + } + + fn joins(&self) -> RecordJoins<'_> { + RecordJoins { + labels: &self.labels, + parent_closed: &self.parent_closed, + created_at: self.created_at.as_deref(), + updated_at: self.updated_at.as_deref(), + custom: Some(&self.custom), + ..RecordJoins::default() + } + } +} + +/// The input of the store-free `list` (6j6v.xkr7): what a reader read for a [`ListReads`]. +/// +/// `#[non_exhaustive]`: build it with [`ListBoard::new`] and set the fields. +#[derive(Debug, Clone, Default)] +#[non_exhaustive] +pub struct ListBoard { + /// The types `items` holds every live, unarchived item of; `None`: every type. + pub types: Option>, + /// The listed items, by id: live and unarchived, of `types`. + pub items: BTreeMap, + /// `next`'s board, for a key that reads the rank of a type `next` ranks + /// ([`ListReads::next`]): the active tickets' rows — their labels and custom values when the + /// graph reads them — and their [`Board`]. Its own `ranking` stays `None`. + pub active: Option, + /// The graph the derived keys read ([`ListReads::ranking`]). + pub ranking: Option, +} + +impl ListBoard { + /// The listed items of `types` (`None`: every type), keyed by their ids. + pub fn new( + types: Option>, + items: impl IntoIterator, + ) -> ListBoard { + ListBoard { + types, + items: items.into_iter().map(|i| (i.item.id.clone(), i)).collect(), + active: None, + ranking: None, + } + } +} + +/// What the store-free `list` `q` reads under `cfg` (6j6v.xkr7): the listed types — `q`'s type +/// filter, else every type —, `next`'s board when a `[ranking.list]` key reads a rank `next` gives +/// and `q` has a `now`, and the graph the derived keys of a `sort = rank` read. It depends on the +/// query, not on the board. +pub fn list_reads(cfg: &PluginConfig, q: &ListQuery) -> ListReads { + derived::reads_for_list(cfg, q) +} + +/// The store's [`ListBoard`] for `reads` — the parity oracle for +/// `nxs_fold_ddb::records::list_board`, as [`active_board`] is for the records: the listed items +/// with their joins read in bulk, `next`'s board from the store's selection, and the graph. +pub fn list_board(store: &Store, reads: &ListReads) -> Result { + let items: Vec = derived::list_scope(store)? + .into_iter() + .filter(|i| { + reads + .types + .as_ref() + .is_none_or(|types| i.item_type.as_deref().is_some_and(|t| types.contains(t))) + }) + .collect(); + let ids: Vec<&str> = items.iter().map(|i| i.id.as_str()).collect(); + let mut labels = store.labels_of_bulk(&ids)?; + let mut custom = store.custom_fields_of_bulk(&ids)?; + let mut ts = store.item_timestamps_of_bulk(&ids)?; + let mut contributes = store.contributes_to_of_bulk(&ids)?; + let mut listed = Vec::with_capacity(items.len()); + for item in items { + let id = item.id.clone(); + let parent_closed = parent_closed_reason(store, &item)?; + let (created_at, updated_at) = ts.remove(&id).unwrap_or_default(); + let mut i = ListItem::new(item); + i.labels = labels.remove(&id).unwrap_or_default(); + i.custom = custom.remove(&id).unwrap_or_default(); + i.created_at = created_at; + i.updated_at = updated_at; + i.contributes_to = contributes.remove(&id).unwrap_or_default(); + i.parent_closed = parent_closed; + listed.push(i); + } + let mut board = ListBoard::new(reads.types.clone(), listed); + let graph_reads = reads.ranking.clone().unwrap_or_default(); + if reads.next { + let selection = nexus_flow_core::graph::select(store.connection())?; + let ids: Vec<&str> = selection.tickets().map(|t| t.id.as_str()).collect(); + let items = store.get_items(&ids)?; + let mut labels = if graph_reads.labels { + store.labels_of_bulk(&ids)? + } else { + BTreeMap::new() + }; + let mut custom = if graph_reads.custom { + store.custom_fields_of_bulk(&ids)? + } else { + BTreeMap::new() + }; + let tickets = items.into_iter().map(|item| { + let id = item.id.clone(); + let mut t = ActiveTicket::new(item); + t.labels = labels.remove(&id).unwrap_or_default(); + t.custom = custom.remove(&id).unwrap_or_default(); + t + }); + board.active = Some(ActiveBoard::new(selection, tickets)); + } + if let Some(r) = &reads.ranking { + let mut held: BTreeSet<&str> = board.items.keys().map(String::as_str).collect(); + if let Some(active) = &board.active { + held.extend(active.tickets.keys().map(String::as_str)); + } + let graph = ranking_graph(store, r, &held)?; + board.ranking = Some(graph); + } + Ok(board) +} + +/// [`list_page`] without a store (6j6v.xkr7): the rows of `board` that `q` keeps — its `status`, +/// its filter through the same membership rule ([`NextFilter`]) — ordered by `q.sort`, with the +/// derived keys of a `sort = rank` from the board's graph, and cut to `q.limit` with the total +/// disclosed. The same rows in the same order as [`list_page`] over a store holding the same board. +/// +/// Not paged: the store path's page cache is machine-local, and a server's paging is an open +/// decision (6j6v.15ed) — as for [`next_active_filtered`], a server cuts this list itself. So +/// `validation` for `q.paginate` or a token; and for a malformed `q.now`, a type filter (or none) +/// the board does not list every item for, and a board read for less than the ranking needs. +pub fn list_board_page(cfg: &PluginConfig, board: &ListBoard, q: &ListQuery) -> Result { + if q.paginate || q.token.is_some() { + return Err(NxfError::validation( + "the store-free `list` is not paged — a server cuts it itself; drop paginate and the \ + token", + )); + } + if let Some(now) = &q.now { + crate::validate::iso_date(now)?; + } + let listed = match &board.types { + None => true, + Some(types) => { + !q.filter.types.is_empty() && q.filter.types.iter().all(|t| types.contains(t)) + } + }; + if !listed { + return Err(NxfError::validation( + "the list board was read for other types than the query lists — read it with \ + `list_reads` for this query", + )); + } + let mut items: Vec = board + .items + .values() + // The board holds live, unarchived items; a caller-built one is held to the same rule. + .filter(|i| i.item.deleted.as_deref() != Some("1") && i.item.archived.is_none()) + .filter(|i| { + q.status + .as_deref() + .is_none_or(|s| i.item.status.as_deref() == Some(s)) + }) + .filter(|i| q.filter.keeps(&i.item, &i.labels, &i.contributes_to)) + .map(|i| i.item.clone()) + .collect(); + let key = q.sort.unwrap_or(DEFAULT_SORT_LIST); + if key == SortKey::Rank { + let derived = derived::compute_list(cfg, board, &items, q.now.as_deref())?; + let rank = TypeRank::list(cfg).with_derived(derived.as_ref()); + items.sort_by(|a, b| rank.cmp(a, b)); + } else { + order_by(cfg, &mut items, key); + } + Ok(truncate_next(items, q.limit)) +} + +/// [`list_board_page`] as the `list` JSON envelope `{"items", "total"}` — the value +/// [`Engine::list_query_value`] gives for the same board and an unpaged query, every record with +/// its joins from the item's [`ListItem`]. +/// +/// [`Engine::list_query_value`]: crate::engine::Engine::list_query_value +pub fn list_board_value(cfg: &PluginConfig, board: &ListBoard, q: &ListQuery) -> Result { + let page = list_board_page(cfg, board, q)?; + let records = Value::Array( + page.items + .iter() + .map(|i| { + let joins = board + .items + .get(&i.id) + .map(ListItem::joins) + .unwrap_or_default(); + list_record(Some(cfg), i, &joins) + }) + .collect(), + ); + Ok(next_page_envelope(records, &page, false)) +} + // ---- lane-ranked search (C6 #916.6) ---------------------------------------- // // `search` is a facade method (not CLI-only) so the CLI and an in-process consumer share ONE diff --git a/crates/facade/src/read/derived.rs b/crates/facade/src/read/derived.rs index 9906834..7c8d8ad 100644 --- a/crates/facade/src/read/derived.rs +++ b/crates/facade/src/read/derived.rs @@ -26,8 +26,18 @@ //! custom values — and runs steps 3 and 4 on the same [`Context`], so the two paths cannot evaluate //! a key apart. A neighbour counts when it is LIVE, closed and archived ones included, on both //! paths (spec §6, owner 2026-10-11). What the graph must hold is [`reads_for_next`]. - -use super::{lanes, mentioners_of, resolve_items, ActiveBoard, RankingGraph, RankingReads}; +//! +//! The store-free `list` (6j6v.xkr7, [`super::list_board_page`]) ranks a [`ListBoard`] the same +//! way: [`compute_list`] takes the listed rows from the board, `next`'s candidates — when a key +//! reads a rank `next` gives — from its active board, and every other row from its graph. A +//! `[ranking.list]` rank a key reads is computed over the type's live items, which the graph holds +//! for every type in [`RankingReads::list_types`] (the `typed` index on a server), so a `next` key +//! may read one too ([`compute_active`]). What a `list` reads is [`reads_for_list`]. + +use super::{ + lanes, mentioners_of, resolve_items, ActiveBoard, ListBoard, ListQuery, ListReads, + RankingGraph, RankingReads, SortKey, +}; use crate::error::{NxfError, Result}; use crate::plugin::{Dir, Nulls, Pick, PluginConfig, RankKey, RankOf, RankSpec, Verb}; use crate::record::{raw_field, RANKABLE_FIELDS}; @@ -138,6 +148,55 @@ pub(super) fn list_scope(store: &Store) -> Result> { .collect()) } +/// A live row and its joins, wherever a store-free board holds it: an active ticket, a listed item, +/// or a neighbour in the graph. +#[derive(Clone, Copy)] +pub(super) struct Joined<'b> { + item: &'b ItemRow, + labels: &'b [String], + custom: &'b BTreeMap, +} + +/// The row `id` of an [`ActiveBoard`] with its `graph`: an active ticket's, else a neighbour's. +fn active_joined<'b>( + board: &'b ActiveBoard, + graph: &'b RankingGraph, + id: &str, +) -> Option> { + match board.tickets.get(id) { + Some(t) => Some(Joined { + item: &t.item, + labels: &t.labels, + custom: &t.custom, + }), + None => graph.neighbours.get(id).map(|n| Joined { + item: &n.item, + labels: &n.labels, + custom: &n.custom, + }), + } +} + +/// The row `id` of a [`ListBoard`] with its `graph`: a listed item's, else an active ticket's, +/// else a neighbour's. +fn list_joined<'b>(board: &'b ListBoard, graph: &'b RankingGraph, id: &str) -> Option> { + if let Some(i) = board.items.get(id) { + return Some(Joined { + item: &i.item, + labels: &i.labels, + custom: &i.custom, + }); + } + match &board.active { + Some(active) => active_joined(active, graph, id), + None => graph.neighbours.get(id).map(|n| Joined { + item: &n.item, + labels: &n.labels, + custom: &n.custom, + }), + } +} + /// The derived side of one query, decided before any read: the type chain the ranked rows' orders /// read the rank of (leaves first), and what the derived keys in play read. struct Plan<'c> { @@ -204,6 +263,57 @@ impl Plan<'_> { fn reads_list(&self) -> bool { self.chain.iter().any(|(v, _)| *v == Verb::List) } + + /// Whether some order of the chain is a `[ranking.next]` order. + fn reads_next(&self) -> bool { + self.chain.iter().any(|(v, _)| *v == Verb::Next) + } + + /// What a [`RankingGraph`] must hold for this plan: its keys' reads, and the live items of + /// every `[ranking.list]` type of the chain the board does not list itself (`held`). + fn reads(&self, held: impl Fn(&str) -> bool) -> RankingReads { + let mut reads = self.needs.reads(); + reads.list_types = self + .chain + .iter() + .filter(|(v, ty)| *v == Verb::List && !held(ty)) + .map(|(_, ty)| ty.clone()) + .collect(); + reads + } +} + +/// The rows of a graph's listed items (the `list` scope of [`RankingReads::list_types`]), looked +/// up through `rows` — every one of them is a live row the board holds. +fn listed_rows<'b>( + graph: &RankingGraph, + rows: impl Fn(&str) -> Option>, +) -> Vec { + graph + .listed + .iter() + .filter_map(|id| rows(id).map(|j| j.item.clone())) + .collect() +} + +/// The graph a store-free query ranks with: the board's, when it was read for `needed`; an empty +/// one when the keys in play read nothing beyond the rows. `validation` otherwise — ranking without +/// those edges would order apart from the store path in silence. +fn graph_for<'g>( + graph: Option<&'g RankingGraph>, + needed: &RankingReads, + empty: &'g RankingGraph, + what: &str, +) -> Result<&'g RankingGraph> { + if *needed == RankingReads::default() { + return Ok(graph.unwrap_or(empty)); + } + graph.filter(|g| g.reads.covers(needed)).ok_or_else(|| { + NxfError::validation(format!( + "{what} ranks by derived ranking keys, and the board was read without the graph they \ + read — fill its `ranking` for the reads its plugin names" + )) + }) } /// Every row a key is evaluated on: the ranked rows and the rows of the chain's scopes. @@ -268,12 +378,13 @@ pub(super) fn compute( /// [`compute`] for the store-free `next` (6j6v.1c5s): the values the derived keys of /// `[ranking.next]` rank `candidates` — every candidate of the board — by, read from the /// [`RankingGraph`] a reader filled instead of from a store. The candidates are `next`'s scope for -/// "rank", as on the store path. `None` when the policy has no derived key. +/// "rank", as on the store path; the `list` scope of a `[ranking.list]` type the chain reaches is +/// the graph's listed items (6j6v.xkr7). `None` when the policy has no derived key. /// /// The same [`Context`] evaluates the keys as on the store path; only where its edges, rows, -/// labels and custom values come from differs ([`Context::read_graph`]). A chain that reaches a -/// `[ranking.list]` order is refused before this is called ([`reads_for_next`]); it answers -/// `validation` here too rather than rank without that scope. +/// labels and custom values come from differs ([`Context::read_graph`]). `validation` when the graph +/// was read for less than the plan needs — a `[ranking.list]` type of the chain missing from +/// [`RankingReads::list_types`] among it. pub(super) fn compute_active( cfg: &PluginConfig, board: &ActiveBoard, @@ -283,84 +394,179 @@ pub(super) fn compute_active( let Some(plan) = plan(cfg, Verb::Next, candidates) else { return Ok(None); }; - if plan.reads_list() { - return Err(list_scope_refusal(cfg)); - } - let node_rows = plan.node_rows(candidates, &[]); + let needed = plan.reads(|_| false); + let graph = graph_for( + Some(graph), + &needed, + graph, + &format!("plugin '{}' (`next`)", cfg.name), + )?; + let rows = |id: &str| active_joined(board, graph, id); + let listed = listed_rows(graph, rows); + let node_rows = plan.node_rows(candidates, &listed); let mut ctx = Context::new(cfg, subjects(candidates, &node_rows)); - ctx.read_graph(board, graph, &plan.needs); + ctx.read_graph(rows, graph, &plan.needs); Ok(Some(ctx.finish(&plan, Verb::Next, candidates, &node_rows))) } -/// What the store-free `next` of `cfg` reads beyond the active tickets' records (6j6v.1c5s): -/// `None` when `[ranking.next]` declares no derived key. Gathered over every order of the section, -/// so it does not depend on which types a board holds. +/// [`compute`] for the store-free `list` (6j6v.xkr7): the values the derived keys of +/// `[ranking.list]` rank `ranked` by, from a [`ListBoard`]. The `list` scope of a type is its live, +/// unarchived items — the board's listed items for the types it lists, the graph's listed items +/// for the others; `next`'s scope, when the chain reaches a `[ranking.next]` order, is the +/// candidates of the board's active board at `now`. `None` when the policy has no derived key. /// -/// `validation` when an order of `[ranking.next]` reads — directly or down its type chain — the -/// rank of a type only `[ranking.list]` orders (spec §12, first point): that rank is computed over -/// the live items of the type, and a server reads only the active ones (owner, 2026-10-11). +/// `validation` as on the store path when the chain reads a `next` rank and `now` is missing, and +/// when the board was read for less than the plan needs: no active board for such a chain, a +/// graph read for fewer reads, a chain type neither listed nor in the graph. +pub(super) fn compute_list( + cfg: &PluginConfig, + board: &ListBoard, + ranked: &[ItemRow], + now: Option<&str>, +) -> Result> { + let Some(plan) = plan(cfg, Verb::List, ranked) else { + return Ok(None); + }; + let held = |ty: &str| board.types.as_ref().is_none_or(|t| t.contains(ty)); + let needed = plan.reads(held); + let empty = RankingGraph::default(); + let graph = graph_for( + board.ranking.as_ref(), + &needed, + &empty, + &format!("plugin '{}' (`list`)", cfg.name), + )?; + let next_owned; + let next_rows: &[ItemRow] = if plan.reads_next() { + let now = now.ok_or_else(|| { + NxfError::validation( + "the plugin's [ranking.list] reads the rank of a type that `next` ranks, which is \ + computed over `next`'s candidates at a reference time — pass `now`", + ) + })?; + let active = board.active.as_ref().ok_or_else(|| { + NxfError::validation(format!( + "plugin '{}' ranks `list` by the rank of a type that `next` ranks, and the board \ + was read without the active board that rank is computed over — read it with \ + `ListReads::next`", + cfg.name + )) + })?; + let candidates = active.board.lanes(now).candidates; + next_owned = active.items(candidates.iter().map(|c| &c.id)); + &next_owned + } else { + &[] + }; + let rows = |id: &str| list_joined(board, graph, id); + let list_owned: Vec = if plan.reads_list() { + let mut scope: Vec = board.items.values().map(|i| i.item.clone()).collect(); + scope.extend( + listed_rows(graph, rows) + .into_iter() + .filter(|i| !board.items.contains_key(&i.id)), + ); + scope + } else { + Vec::new() + }; + let node_rows = plan.node_rows(next_rows, &list_owned); + let mut ctx = Context::new(cfg, subjects(ranked, &node_rows)); + ctx.read_graph(rows, graph, &plan.needs); + Ok(Some(ctx.finish(&plan, Verb::List, ranked, &node_rows))) +} + +/// What the store-free `next` of `cfg` reads beyond the active tickets' records (6j6v.1c5s): +/// `None` when `[ranking.next]` declares no derived key. Gathered over every order of the section +/// and of its type chain, so it does not depend on which types a board holds. Since 6j6v.xkr7 an +/// order that reads the rank of a type only `[ranking.list]` orders is no longer refused: the +/// type lands in [`RankingReads::list_types`], whose live items a reader puts in the graph. pub(super) fn reads_for_next(cfg: &PluginConfig) -> Result> { - let ranking = &cfg.ranking; - let spec = ranking.spec(Verb::Next); + let spec = cfg.ranking.spec(Verb::Next); if !spec.has_derived() { return Ok(None); } + let orders: Vec<&[RankKey]> = std::iter::once(spec.order.as_slice()) + .chain(spec.by_type.values().map(|o| o.order.as_slice())) + .collect(); + // `next`'s own per-type orders start the chain: an order there may read another's rank. + let starts = spec.by_type.keys().map(|ty| (Verb::Next, ty.clone())); + Ok(Some(chain_reads(cfg, &orders, starts, |_| false).0)) +} + +/// What the store-free `list` `q` asks for reads (6j6v.xkr7): its listed types (`None`: every +/// type), whether `next`'s board is needed, and the graph. Only `sort = rank` ranks by the +/// derived keys; any other order reads the listed items alone. +pub(super) fn reads_for_list(cfg: &PluginConfig, q: &ListQuery) -> ListReads { + let types: Option> = + (!q.filter.types.is_empty()).then(|| q.filter.types.iter().cloned().collect()); + let mut reads = ListReads { + types: types.clone(), + next: false, + ranking: None, + }; + let spec = cfg.ranking.spec(Verb::List); + if q.sort != Some(SortKey::Rank) || !spec.has_derived() { + return reads; + } + let orders: Vec<&[RankKey]> = match &types { + Some(types) => types + .iter() + .map(|ty| spec.order_for(Some(ty.as_str()))) + .collect(), + None => std::iter::once(spec.order.as_slice()) + .chain(spec.by_type.values().map(|o| o.order.as_slice())) + .collect(), + }; + let starts: Vec = orders + .iter() + .flat_map(|keys| cfg.ranking.rank_deps(keys, Verb::List)) + .collect(); + let held = |ty: &str| types.as_ref().is_none_or(|t| t.contains(ty)); + let (graph, reads_next) = chain_reads(cfg, &orders, starts, held); + // Without `now` a chain that reads a `next` rank answers `validation` where a ranked row + // needs it, as on the store path — reading `next`'s board for it would be wasted. + reads.next = reads_next && q.now.is_some(); + if graph != RankingReads::default() { + reads.ranking = Some(graph); + } + reads +} + +/// The reads of `orders` and of every order down the type chain from `starts`, the chain's +/// `[ranking.list]` types not `held` among them, and whether that chain holds a `[ranking.next]` +/// order. +fn chain_reads( + cfg: &PluginConfig, + orders: &[&[RankKey]], + starts: impl IntoIterator, + held: impl Fn(&str) -> bool, +) -> (RankingReads, bool) { let mut chain: Vec = Vec::new(); let mut seen: BTreeSet = BTreeSet::new(); - for ty in spec.by_type.keys() { - post_order(cfg, (Verb::Next, ty.clone()), &mut seen, &mut chain); - } - if chain.iter().any(|(v, _)| *v == Verb::List) { - return Err(list_scope_refusal(cfg)); + for node in starts { + post_order(cfg, node, &mut seen, &mut chain); } - let keys: Vec<&RankKey> = spec - .order + let mut keys: Vec<&RankKey> = orders .iter() - .chain(spec.by_type.values().flat_map(|o| o.order.iter())) + .flat_map(|o| o.iter()) .filter(|k| k.is_derived()) .collect(); - let needs = Needs::of(&keys); - let mut reads = RankingReads::default(); - reads.kinds.extend(needs.kinds.iter().copied()); - if needs.mentioned_by { - reads.kinds.insert(EdgeKind::Mentions); + for (v, ty) in &chain { + keys.extend( + cfg.ranking.spec(*v).by_type[ty] + .order + .iter() + .filter(|k| k.is_derived()), + ); } - reads.labels = needs.labels; - reads.custom = needs.custom; - Ok(Some(reads)) -} - -/// The refusal of a `[ranking.next]` whose type chain reaches a `[ranking.list]` order, on the -/// store-free path. -fn list_scope_refusal(cfg: &PluginConfig) -> NxfError { - let ranking = &cfg.ranking; - let reached: Vec = ranking - .spec(Verb::Next) - .by_type - .keys() - .flat_map(|ty| { - let mut chain = Vec::new(); - post_order( - cfg, - (Verb::Next, ty.clone()), - &mut BTreeSet::new(), - &mut chain, - ); - chain - .into_iter() - .filter(|(v, _)| *v == Verb::List) - .map(move |(_, list_ty)| { - format!("[ranking.next.{ty}] reads the rank of '{list_ty}'") - }) - }) + let mut reads = Needs::of(&keys).reads(); + reads.list_types = chain + .iter() + .filter(|(v, ty)| *v == Verb::List && !held(ty)) + .map(|(_, ty)| ty.clone()) .collect(); - NxfError::validation(format!( - "plugin '{}' ranks `next` by the rank of a type only [ranking.list] orders ({}): that rank \ - is computed over the type's live items, which the store-free lanes do not hold — answer \ - `next` from a store", - cfg.name, - reached.join("; ") - )) + (reads, chain.iter().any(|(v, _)| *v == Verb::Next)) } /// Depth-first post-order over the type chain: every order a node reads the rank of lands before @@ -419,6 +625,21 @@ impl Needs { } } +impl Needs { + /// These needs as the [`RankingReads`] of a graph: `mentions` among the kinds when a count reads + /// `mentioned_by`. No list types; the caller adds them. + fn reads(&self) -> RankingReads { + let mut reads = RankingReads::default(); + reads.kinds.extend(self.kinds.iter().copied()); + if self.mentioned_by { + reads.kinds.insert(EdgeKind::Mentions); + } + reads.labels = self.labels; + reads.custom = self.custom; + reads + } +} + /// One count, from a `count` key or a `sum` summand. struct Count<'a> { kind: &'a str, @@ -541,11 +762,16 @@ impl<'a> Context<'a> { /// The reads from a [`RankingGraph`] instead of a store (6j6v.1c5s): the same adjacency, reverse /// mentions, neighbour rows, labels and custom values [`read_edges`](Self::read_edges) reads, - /// taken from the graph's edges and from the board's tickets and the graph's neighbours. The - /// graph holds every edge of the kinds read that touches an active ticket with both ends live, - /// `parent` as the `present_parent` projection — the edges the store's pass yields for subjects - /// that are all active tickets, as `next`'s candidates are. - fn read_graph(&mut self, board: &ActiveBoard, graph: &RankingGraph, needs: &Needs) { + /// taken from the graph's edges and, through `rows`, from the board's rows — its active + /// tickets, its listed items — and the graph's neighbours. The graph holds every edge of the + /// kinds read that touches a subject with both ends live, `parent` as the `present_parent` + /// projection — the edges the store's pass yields for the same subjects. + fn read_graph<'b>( + &mut self, + rows: impl Fn(&str) -> Option>, + graph: &RankingGraph, + needs: &Needs, + ) { let mut mention_sources: BTreeMap> = BTreeMap::new(); for e in &graph.edges { if needs.kinds.contains(&e.kind) { @@ -572,21 +798,13 @@ impl<'a> Context<'a> { .insert(e.from.as_str()); } } - // A row of the board: an active ticket's, or a live neighbour's. - let board_row = |id: &str| -> Option<&ItemRow> { - board - .tickets - .get(id) - .map(|t| &t.item) - .or_else(|| graph.neighbours.get(id).map(|n| &n.item)) - }; for (target, froms) in mention_sources { - let rows: Vec = froms + let found: Vec = froms .into_iter() - .filter_map(|f| board_row(f).cloned()) + .filter_map(|f| rows(f).map(|j| j.item.clone())) .collect(); - if !rows.is_empty() { - self.mentioners.insert(target, rows); + if !found.is_empty() { + self.mentioners.insert(target, found); } } let missing: BTreeSet<&str> = self @@ -599,23 +817,19 @@ impl<'a> Context<'a> { .collect(); self.neighbours = missing .into_iter() - .filter_map(|id| board_row(id).map(|r| (id.to_string(), r.clone()))) + .filter_map(|id| rows(id).map(|j| (id.to_string(), j.item.clone()))) .collect(); if needs.labels || needs.custom { let mut ids: BTreeSet<&str> = self.neighbours.keys().map(String::as_str).collect(); ids.extend(self.subjects.keys().copied()); ids.extend(self.mentioners.values().flatten().map(|i| i.id.as_str())); for id in ids { - let (labels, custom) = match (board.tickets.get(id), graph.neighbours.get(id)) { - (Some(t), _) => (&t.labels, &t.custom), - (None, Some(n)) => (&n.labels, &n.custom), - (None, None) => continue, - }; - if needs.labels && !labels.is_empty() { - self.labels.insert(id.to_string(), labels.clone()); + let Some(j) = rows(id) else { continue }; + if needs.labels && !j.labels.is_empty() { + self.labels.insert(id.to_string(), j.labels.to_vec()); } - if needs.custom && !custom.is_empty() { - self.custom.insert(id.to_string(), custom.clone()); + if needs.custom && !j.custom.is_empty() { + self.custom.insert(id.to_string(), j.custom.clone()); } } } diff --git a/crates/facade/tests/derived_ranking.rs b/crates/facade/tests/derived_ranking.rs index cdb778c..c49f900 100644 --- a/crates/facade/tests/derived_ranking.rs +++ b/crates/facade/tests/derived_ranking.rs @@ -11,6 +11,8 @@ //! - Determinism (§9): another edge insertion order ranks alike, and ties fall to the id. //! - Bounded cost (§9, a query-shape guard in the style of xn8s): the edge pass runs once per query //! however many candidates there are. +//! - Without a store (6j6v.1c5s, 6j6v.xkr7): `read::next_active` and `read::list_board_page` over +//! the boards a server reads rank as the store path does. //! //! Its own test binary, so the fixture plugin does not pollute the facade lib tests' "exactly two //! OSS plugins" assertion. @@ -521,11 +523,12 @@ fn a_closed_archived_or_deleted_neighbour_reads_alike_with_and_without_a_store() } #[test] -fn a_next_order_reading_the_rank_of_a_type_only_list_orders_is_refused_without_a_store() { +fn a_next_order_reading_the_rank_of_a_type_only_list_orders_ranks_from_the_listed_items() { // Spec §12 point 1 lets a `next` key read the rank `[ranking.list]` gives a type. That rank is - // computed over the type's live items, which a board of active tickets does not hold: the - // store-free `next` refuses the plugin (owner, 2026-10-11), the store path ranks it. Any other - // derived `next` key ranks without a store. + // computed over the type's live, unarchived items, which a board of active tickets does not + // hold: since 6j6v.xkr7 the reads name the type (`list_types`) and the graph holds its items — + // a server reads them off its `typed` index — so the store-free `next` ranks as the store + // path does. A board read without them is refused. let cfg: plugin::PluginConfig = toml::from_str( r#" name = "list-rank-fixture" @@ -533,7 +536,7 @@ fn a_next_order_reading_the_rank_of_a_type_only_list_orders_is_refused_without_a en = "x" de = "y" [priority] - labels = ["P0", "P1"] + labels = ["P0", "P1", "P2"] [types] list = ["action", "person"] [vocabulary.status] @@ -545,33 +548,188 @@ fn a_next_order_reading_the_rank_of_a_type_only_list_orders_is_refused_without_a [ranking.next.action] order = [ { related = "mentions", type = "person", of = "rank", nulls = "last" } ] [ranking.list.person] - order = [ { field = "priority", dir = "asc" } ] + order = [ { count = "mentioned_by", dir = "desc" }, { field = "priority", dir = "asc" } ] [presentation.list] columns = ["id"] "#, ) .unwrap(); let mut s = Store::open_in_memory(1); - item(&mut s, "zz12.a1", "action", "1"); - item(&mut s, "zz12.u1", "person", "0"); - edge(&mut s, "zz12.a1", "zz12.u1", EdgeKind::Mentions); - let err = read::next_ranking_reads(&cfg).unwrap_err(); - assert_eq!(err.kind, ErrorKind::Validation); - assert!( - err.msg - .contains("[ranking.next.action] reads the rank of 'person'"), - "{}", - err.msg + // Persons: u1 mentioned twice (rank 0); u2 and the closed u3 once each, u3 with the better + // priority; u4 archived (no rank: outside the list scope); u5 deleted. + for (u, prio) in [ + ("u1", "2"), + ("u2", "1"), + ("u3", "0"), + ("u4", "0"), + ("u5", "0"), + ] { + item(&mut s, &format!("zz12.{u}"), "person", prio); + } + close(&mut s, "zz12.u3"); + s.set_field("zz12.u4", "archived", Some("2026-06-01".into()), "t"); + for (a, person) in [ + ("zz12.a1", "zz12.u2"), + ("zz12.a2", "zz12.u1"), + ("zz12.a3", "zz12.u3"), + ("zz12.a4", "zz12.u4"), + ("zz12.a5", "zz12.u5"), + ] { + item(&mut s, a, "action", "1"); + edge(&mut s, a, person, EdgeKind::Mentions); + } + item(&mut s, "zz12.a6", "action", "1"); + edge(&mut s, "zz12.a6", "zz12.u1", EdgeKind::Mentions); + s.delete_item("zz12.u5", "t"); + + let reads = read::next_ranking_reads(&cfg) + .unwrap() + .expect("derived next keys"); + assert_eq!( + reads.list_types.iter().collect::>(), + ["person"], + "{reads:?}" ); - let board = read::active_board(&s).unwrap(); + let board = read::with_ranking(&s, read::active_board(&s).unwrap(), &reads).unwrap(); + let graph = board.ranking.as_ref().unwrap(); + assert_eq!( + graph.listed.iter().collect::>(), + ["zz12.u1", "zz12.u2", "zz12.u3"], + "the live, unarchived persons" + ); + let store = read::next(&cfg, &s, NOW, None).unwrap(); + assert_eq!( + of_type(&store, "action"), + ["zz12.a2", "zz12.a6", "zz12.a3", "zz12.a1", "zz12.a4", "zz12.a5"] + ); + assert_eq!( + ids(&read::next_active(&cfg, &board, NOW).unwrap()), + ids(&store) + ); + + // Without the person's items the graph is short of what the keys read: refused. + let mut short = reads.clone(); + short.list_types.clear(); + let board = read::with_ranking(&s, read::active_board(&s).unwrap(), &short).unwrap(); assert_eq!( read::next_active(&cfg, &board, NOW).unwrap_err().kind, ErrorKind::Validation ); +} + +#[test] +fn the_store_free_list_ranks_persons_as_list_query_does() { + // 6j6v.xkr7: the hosted `list` — `read::list_board_page` over a `ListBoard` read for + // `read::list_reads` — gives the rows, the order, the total and the JSON `Engine::list_query` + // gives: persons by a sum of mentions, then by the rank `next` gives the projects they + // contribute to (`now` takes `next`'s candidates), filtered by type, status, container and + // label as locally. + let tmp = workspace("derived-rank-fixture"); + spec_board(tmp.path()); + let cfg = plugin::load("derived-rank-fixture").unwrap(); + let s = writer(tmp.path()); + let engine = engine(&tmp); + let queries = [ + ListQuery::new() + .sort(SortKey::Rank) + .filter(NextFilter::new().with_type("person")) + .now(NOW), + ListQuery::new() + .sort(SortKey::Rank) + .filter(NextFilter::new().with_type("person")) + .now(NOW) + .limit(2), + ListQuery::new().sort(SortKey::Rank).now(NOW), + ListQuery::new() + .sort(SortKey::Rank) + .filter( + NextFilter::new() + .with_type("action") + .with_container("zz12.p5"), + ) + .now(NOW), + ListQuery::new() + .sort(SortKey::Rank) + .filter(NextFilter::new().with_label("waiting")) + .now(NOW), + ListQuery::new().status("closed"), + ListQuery::new() + .filter(NextFilter::new().with_type("project").with_type("goal")) + .sort(SortKey::Rank), + ]; + for q in &queries { + let reads = read::list_reads(&cfg, q); + let board = read::list_board(&s, &reads).unwrap(); + let page = read::list_board_page(&cfg, &board, q).unwrap(); + let want = engine.list_query(q).unwrap(); + assert_eq!(ids(&page.items), ids(&want.items), "{q:?}"); + assert_eq!(page.total, want.total, "{q:?}"); + assert_eq!( + read::list_board_value(&cfg, &board, q).unwrap(), + engine.list_query_value(q).unwrap(), + "{q:?}" + ); + } + let persons = read::list_board_page( + &cfg, + &read::list_board(&s, &read::list_reads(&cfg, &queries[0])).unwrap(), + &queries[0], + ) + .unwrap(); + assert_eq!(ids(&persons.items), PERSONS); + let reads = read::list_reads(&cfg, &queries[0]); assert!( - read::next(&cfg, &s, NOW, None).is_ok(), - "the store path ranks it" + reads.next, + "persons read the rank `next` gives projects: {reads:?}" ); + assert_eq!( + reads.types, + Some(["person".to_string()].into_iter().collect()) + ); +} + +#[test] +fn the_store_free_list_refuses_what_its_board_was_not_read_for() { + let tmp = workspace("derived-rank-fixture"); + spec_board(tmp.path()); + let cfg = plugin::load("derived-rank-fixture").unwrap(); + let s = writer(tmp.path()); + let persons = ListQuery::new() + .sort(SortKey::Rank) + .filter(NextFilter::new().with_type("person")) + .now(NOW); + let board = read::list_board(&s, &read::list_reads(&cfg, &persons)).unwrap(); + let refused = |q: &ListQuery, board: &read::ListBoard| { + let err = read::list_board_page(&cfg, board, q).unwrap_err(); + assert_eq!(err.kind, ErrorKind::Validation, "{q:?}"); + err.msg + }; + // Not paged: a server cuts the list itself. + refused(&persons.clone().limit(2).paginate(), &board); + // Other types, or every type, than the board lists. + refused( + &ListQuery::new().filter(NextFilter::new().with_type("project")), + &board, + ); + refused(&ListQuery::new(), &board); + // Without `now`, `next`'s candidates cannot be taken — as on the store path. + let no_now = ListQuery::new() + .sort(SortKey::Rank) + .filter(NextFilter::new().with_type("person")); + let reads = read::list_reads(&cfg, &no_now); + assert!(!reads.next, "nothing to read `next`'s board for: {reads:?}"); + let msg = refused(&no_now, &read::list_board(&s, &reads).unwrap()); + assert!(msg.contains("pass `now`"), "{msg}"); + // A board without the graph its keys read. + let mut short = read::list_reads(&cfg, &persons); + short.ranking = None; + let msg = refused(&persons, &read::list_board(&s, &short).unwrap()); + assert!(msg.contains("derived ranking keys"), "{msg}"); + // Without `next`'s board. + let mut short = read::list_reads(&cfg, &persons); + short.next = false; + let msg = refused(&persons, &read::list_board(&s, &short).unwrap()); + assert!(msg.contains("active board"), "{msg}"); } // ---- each kind over hand-built boards (§9 "per kind") ------------------------------------- diff --git a/crates/fold-ddb/src/board.rs b/crates/fold-ddb/src/board.rs index 36aea0e..d43f5d0 100644 --- a/crates/fold-ddb/src/board.rs +++ b/crates/fold-ddb/src/board.rs @@ -1,5 +1,5 @@ -//! The board's two indexes, kept beside the folded rows, and the reads that use them -//! (6j6v.vvw6 points 4 and 5). +//! The board's indexes, kept beside the folded rows, and the reads that use them +//! (6j6v.vvw6 points 4 and 5; `typed`, 6j6v.xkr7). //! //! # `active` //! @@ -27,6 +27,17 @@ //! one instant keep one order. A ticket without the instant sits under `"0"`, which sorts after //! every date — last, as the local lanes put it. [`dated`] reads one lane newest first. //! +//! # `typed` +//! +//! Sparse as well (6j6v.xkr7): a live item — not deleted, archived or not — with a type sits in +//! partition `#` ([`typed_partition`]) under its own sort key, so ONE Query lists the +//! live items of a type in id order: the `list` scope of a type the derived ranking keys rank by +//! (docs/specs/derived-ranking-keys.md §6). Like the `active` flag of a ticket it is a function of +//! the row alone, set whenever an op touches the row ([`refresh_item`]) — after the row, so a run +//! that dies in between sets it when it folds the op again. Generic: every typed item is listed, +//! whatever the plugin ranks, at one attribute per item row. A row's `typed` attribute never +//! names another row, so a retype or a delete moves nothing else. +//! //! # Who owns a label or a thread link //! //! `label_adds` and `thread_link_adds` key on the add's tag alone, so nothing in them is found by @@ -72,7 +83,8 @@ use crate::layout::{ adjacency_key, adjacency_prefix, contributes_adjacency_key, label_adjacency_key, - link_adjacency_key, row_key, table_prefix, ACTIVE, DATED, DATED_AT, MAX_KEY_BYTES, SK, + link_adjacency_key, row_key, table_prefix, typed_partition, ACTIVE, DATED, DATED_AT, + MAX_KEY_BYTES, SK, TYPED, }; use crate::table::{text, Dated, Row, Table}; use crate::write::{Cond, Write}; @@ -233,6 +245,29 @@ pub fn is_active(row: &Row) -> bool { ) } +/// The type a stored `items` row is listed under on the `typed` index (6j6v.xkr7): its type, when +/// it has one and is live — not deleted; archived or not. A type too long for an index partition +/// ([`MAX_PARTITION_BYTES`](crate::layout::MAX_PARTITION_BYTES)) is never folded: [`typed_partitions`] refuses its op first. +pub fn typed(row: &Row) -> Option<&str> { + if text(row, "deleted") == Some("1") { + return None; + } + text(row, "type") +} + +/// The `typed` partitions the item rows these changes write would be listed under — what +/// [`Folder`](crate::fold::Folder) checks against +/// [`MAX_PARTITION_BYTES`](crate::layout::MAX_PARTITION_BYTES) before an op writes +/// anything. +pub(crate) fn typed_partitions(stream: &str, changes: &[Change]) -> Vec { + changes + .iter() + .filter(|c| c.table == "items") + .filter_map(|c| effect_text(c, "type")) + .map(|ty| typed_partition(stream, ty)) + .collect() +} + /// The `dated` lane of a stored `items` row, and the instant it sorts by. pub fn lane(row: &Row) -> Option<(Dated, String)> { if text(row, "deleted") == Some("1") { @@ -331,6 +366,14 @@ pub(crate) async fn refresh_item(table: &T, id: &str) -> Result<(), T: remove.push(ACTIVE.to_string()); } } + let typed_now = typed(&row).map(|ty| typed_partition(table.stream(), ty)); + let typed_was = text(&row, TYPED).map(str::to_string); + if typed_now != typed_was { + match typed_now { + Some(p) => set.push((TYPED.to_string(), Cell::Text(p))), + None => remove.push(TYPED.to_string()), + } + } let dated_now = lane(&row).map(|(l, at)| (l.partition(table.stream()), dated_key(&at, id))); let dated_was = match (row.get(DATED), row.get(DATED_AT)) { (Some(Cell::Text(p)), Some(Cell::Text(at))) => Some((p.clone(), at.clone())), diff --git a/crates/fold-ddb/src/ddb.rs b/crates/fold-ddb/src/ddb.rs index 6089713..323d715 100644 --- a/crates/fold-ddb/src/ddb.rs +++ b/crates/fold-ddb/src/ddb.rs @@ -1,14 +1,16 @@ //! [`DynamoDbTable`]: one stream's partition of the views table in DynamoDB, and [`create_table`], //! the table the contract in the crate doc describes. //! -//! Every request names the stream's partition key; the board's two reads name an index. There is +//! Every request names the stream's partition key; the board's three index reads name an index. There is //! no Scan anywhere in this file, and the `dynamodb-local` CI job records every request a fold //! makes to prove it stays that way. //! //! The adapter is async and runs on the caller's runtime — a fold run in a Lambda awaits it //! directly, unlike the relay's op store, which bridges the SDK into its synchronous traits. -use crate::layout::{ACTIVE, ACTIVE_INDEX, DATED, DATED_AT, DATED_INDEX, PK, SK}; +use crate::layout::{ + typed_partition, ACTIVE, ACTIVE_INDEX, DATED, DATED_AT, DATED_INDEX, PK, SK, TYPED, TYPED_INDEX, +}; use crate::table::{Dated, Outcome, Row, Table}; use crate::write::Write; use aws_sdk_dynamodb::error::{DisplayErrorContext, ProvideErrorMetadata}; @@ -37,8 +39,8 @@ fn err(e: E) -> DdbError { DdbError(DisplayErrorContext(e).to_string()) } -/// The views table of `name`, as the deploying stack must provision it: keys, both sparse -/// indexes projecting every attribute, on-demand billing. Production code never sends it; the +/// The views table of `name`, as the deploying stack must provision it: keys, the three sparse +/// indexes (`active`, `dated`, `typed`) projecting every attribute, on-demand billing. Production code never sends it; the /// integration tests do, and a stack is checked against it. pub fn create_table(client: &Client, name: &str) -> CreateTableFluentBuilder { let s = |n: &str| { @@ -77,10 +79,12 @@ pub fn create_table(client: &Client, name: &str) -> CreateTableFluentBuilder { .attribute_definitions(s(ACTIVE)) .attribute_definitions(s(DATED)) .attribute_definitions(s(DATED_AT)) + .attribute_definitions(s(TYPED)) .key_schema(key(PK, KeyType::Hash)) .key_schema(key(SK, KeyType::Range)) .global_secondary_indexes(index(ACTIVE_INDEX, ACTIVE, SK)) .global_secondary_indexes(index(DATED_INDEX, DATED, DATED_AT)) + .global_secondary_indexes(index(TYPED_INDEX, TYPED, SK)) } /// One stream's partition of the views table `table`. @@ -303,4 +307,19 @@ impl Table for DynamoDbTable { ) .await } + + async fn query_typed(&self, item_type: &str) -> Result, DdbError> { + self.query_all( + Some(TYPED_INDEX), + "#t = :t", + &[("#t", TYPED)], + &[( + ":t", + AttributeValue::S(typed_partition(&self.stream, item_type)), + )], + false, + None, + ) + .await + } } diff --git a/crates/fold-ddb/src/fold.rs b/crates/fold-ddb/src/fold.rs index e4f8075..9770de4 100644 --- a/crates/fold-ddb/src/fold.rs +++ b/crates/fold-ddb/src/fold.rs @@ -23,7 +23,9 @@ //! so its watermark names no index revision at all and the next run refolds it. Revision 2 //! (6j6v.15ed) adds the `.cadj#` entries, a ticket's outgoing `contributes_to` edges. Revision 3 //! (6j6v.1c5s) makes the `.adj#` entries self-describing — ends, kind, coordinate, presence — for -//! the derived ranking keys ([`crate::records::with_ranking`]). +//! the derived ranking keys ([`crate::records::with_ranking`]), and sets the `typed` index +//! attribute on every live item row (6j6v.xkr7, [`crate::records::list_board`]). Both ship in one +//! release: no stream was ever folded by revision 3 without the attribute. //! //! # What the server cannot see //! @@ -34,7 +36,9 @@ //! coordinate. use crate::board; -use crate::layout::{Layout, LayoutError, MAX_ITEM_BYTES, MAX_KEY_BYTES, WATERMARK_KEY}; +use crate::layout::{ + Layout, LayoutError, MAX_ITEM_BYTES, MAX_KEY_BYTES, MAX_PARTITION_BYTES, WATERMARK_KEY, +}; use crate::table::{Outcome, Table}; use crate::write::{plan, Cond, Write}; use nxs_foundation::change::{Cell, Change, Wins}; @@ -49,7 +53,7 @@ pub const INDEX_DOMAIN: &str = "index"; /// The revision of those rules. 1: the entries under a ticket for its label and thread link adds /// (6j6v.t1ym). 2: the entries under a ticket for its outgoing `contributes_to` edges (6j6v.15ed). /// 3: the adjacency entries copy their edge's ends, kind and coordinate and say whether it is -/// removed (6j6v.1c5s). +/// removed (6j6v.1c5s), and every live item row carries its `typed` index partition (6j6v.xkr7). pub const INDEX_REVISION: i64 = 3; /// The platform's reducers — the board's, the chat's and the memory's. @@ -162,7 +166,8 @@ impl Folder { /// Fold one op: carry out its changes, then keep the board's indexes. /// - /// An op the table cannot hold — a key past `MAX_KEY_BYTES`, a row past `MAX_ITEM_BYTES` — is + /// An op the table cannot hold — a key past `MAX_KEY_BYTES`, a type whose `typed` index + /// partition is past `MAX_PARTITION_BYTES`, a row past `MAX_ITEM_BYTES` — is /// [`Folded::Refused`] before it writes anything, rather than failing the batch for ever: the /// relay keeps it, as it keeps any op a server does not fold. A row that only outgrows the limit /// through earlier ops refuses the one write that would break it ([`Outcome::TooLarge`]); the @@ -197,6 +202,16 @@ impl Folder { ))); } } + if let Some(p) = board::typed_partitions(table.stream(), &changes) + .into_iter() + .find(|p| p.len() > MAX_PARTITION_BYTES) + { + return Ok(refuse(format!( + "a type whose `typed` index partition has {} bytes, past the {MAX_PARTITION_BYTES} \ + a partition key may have", + p.len() + ))); + } if let Some(w) = writes.iter().find(|w| w.bytes() > MAX_ITEM_BYTES) { return Ok(refuse(format!( "a row of {} bytes, past the {MAX_ITEM_BYTES} an item may have", diff --git a/crates/fold-ddb/src/layout.rs b/crates/fold-ddb/src/layout.rs index ffd4943..73475e6 100644 --- a/crates/fold-ddb/src/layout.rs +++ b/crates/fold-ddb/src/layout.rs @@ -26,18 +26,26 @@ pub const ACTIVE: &str = "nxf_active"; pub const DATED: &str = "nxf_dated"; /// The sort key of the `dated` index: the instant the ticket entered its lane. pub const DATED_AT: &str = "nxf_dated_at"; +/// The partition key of the `typed` index (6j6v.xkr7): `#` ([`typed_partition`]), +/// present only on a live item row — not deleted, archived or not — that has a type. +pub const TYPED: &str = "nxf_typed"; /// The `active` index. pub const ACTIVE_INDEX: &str = "active"; /// The `dated` index. pub const DATED_INDEX: &str = "dated"; +/// The `typed` index (6j6v.xkr7). +pub const TYPED_INDEX: &str = "typed"; /// The longest sort key — and `dated` index sort key — DynamoDB stores, in bytes. pub const MAX_KEY_BYTES: usize = 1024; +/// The longest partition key value DynamoDB stores — on the table or on an index — in bytes. Only +/// the `typed` index's can come near it: it carries an item's type ([`typed_partition`]). +pub const MAX_PARTITION_BYTES: usize = 2048; /// The largest item DynamoDB stores, in bytes, counting attribute names and values. pub const MAX_ITEM_BYTES: usize = 400 * 1024; /// Attribute names no view column may take: the table's own keys and the index attributes. -pub const RESERVED: [&str; 5] = [PK, SK, ACTIVE, DATED, DATED_AT]; +pub const RESERVED: [&str; 6] = [PK, SK, ACTIVE, DATED, DATED_AT, TYPED]; /// The fold's own bookkeeping, outside every view table: a view table's name is an identifier and /// cannot start with `.`. @@ -69,6 +77,13 @@ pub fn row_key(table: &str, key: &[(String, Cell)]) -> Result#`, the +/// type with `%` and `#` escaped as a key value is, so two streams' types can never share a +/// partition (`a` + `b#c` and `a#b` + `c`). +pub fn typed_partition(stream: &str, item_type: &str) -> String { + format!("{stream}#{}", escape(item_type)) +} + /// The prefix every row of `table` starts with. pub fn table_prefix(table: &str) -> String { format!("{table}#") diff --git a/crates/fold-ddb/src/lib.rs b/crates/fold-ddb/src/lib.rs index f97134f..05e279d 100644 --- a/crates/fold-ddb/src/lib.rs +++ b/crates/fold-ddb/src/lib.rs @@ -1,7 +1,8 @@ //! Folding on a server (6j6v.vvw6): the reducers' [described changes](nxs_foundation::change) //! carried out as conditional writes against one table per deployment, the board's `active` and //! `dated` indexes kept beside the rows, next/blocked derived by the same library a local replica -//! uses, and snapshots that are tables plus watermark, without the log. +//! uses, and snapshots that are tables plus watermark, without the log; a type's live items listed +//! by a third index, `typed`, for a ranked hosted `list`. //! //! The pieces, in the order a fold run uses them: [`fold::Folder`] maps each op through the //! platform's reducers and [`write::plan`] turns every change into a [`write::Write`]; @@ -22,27 +23,42 @@ //! | sort key | `sk` | S | `#`, `.adj#`/`.ladj#`/`.tadj##`, `.meta#watermark` | //! | GSI `active` | `nxf_active` / `sk` | S/S | sparse: active tickets and the present dep/parent edges touching one | //! | GSI `dated` | `nxf_dated` / `nxf_dated_at` | S/S | sparse: `#closed` and `#archived`, by `␟` | +//! | GSI `typed` | `nxf_typed` / `sk` | S/S | sparse: `#` on every live item row (not deleted; archived included) | //! //! The `.adj##` entries are plain rows in the stream's partition, on no index. Since //! index revision 3 (6j6v.1c5s) each also carries its edge's `from_id`, `to_id`, `kind`, `lamport`, -//! `site` and a `removed` flag — attributes, not keys: the contract above is unchanged. +//! `site` and a `removed` flag — attributes, not keys. //! -//! Both indexes project ALL attributes, so a lane is one Query and no follow-up reads. The lanes +//! Every index projects ALL attributes, so a lane is one Query and no follow-up reads. +//! +//! **The `typed` index (6j6v.xkr7)** is the one change to this contract since `active` and `dated` +//! (docs/specs/derived-ranking-keys.md §6, owner decision §10.4): one Query lists the live items of +//! a type, the `list` scope a hosted `list` and the derived ranking keys rank over +//! ([`records::list_board`]). Its partition key `nxf_typed` is `#`, the type escaped +//! as a key value is (`%` → `%25`, `#` → `%23`; [`layout::typed_partition`]); its sort key is the +//! table's `sk`. The fold sets it on every item row whose type is set and which is not deleted — +//! closed and archived ones included — and removes it on a delete, written after the row like the +//! other index attributes ([`board`], "`typed`"). The deploying stack adds the GSI FIRST: a fold +//! against a table without it still writes the attribute, which costs nothing and breaks nothing; +//! only the read — a `list` through [`records::list_board`], a `next` whose keys read a +//! `[ranking.list]` rank — needs the index. A type whose partition would pass DynamoDB's 2,048 +//! bytes for an index key is refused at fold time ([`layout::MAX_PARTITION_BYTES`]). The lanes //! need nothing else; the RECORDS a lane is shown with carry joins from other tables, which //! [`records`] reads with a fixed number of requests per active ticket. Billing mode //! is the deployment's call. No local secondary index: without one, DynamoDB sets no limit on the //! size of a stream's partition (the 10 GB item-collection limit applies only to tables with an LSI). //! What does bound a stream is the throughput of ONE partition key: a stream's `active` and `dated` -//! entries each share one index partition — a write-heavy stream is limited to that partition's -//! throughput (1,000 writes per second). +//! entries each share one index partition, its items of one type one `typed` partition — a +//! write-heavy stream is limited to that partition's throughput (1,000 writes per second). //! //! Every row of view table `t` carries its columns as attributes of the same name ([`layout`]); a //! NULL cell is DynamoDB's NULL, an integer a number, text a string. //! //! # Freshness //! -//! A row read (`GetItem`, a Query on the table) is strongly consistent. The two index reads — -//! `board::select` and `board::dated` — are eventually consistent, as every DynamoDB GSI read is: +//! A row read (`GetItem`, a Query on the table) is strongly consistent. The index reads — +//! `board::select`, `board::dated` and the `typed` Query of [`records::list_board`] — are eventually +//! consistent, as every DynamoDB GSI read is: //! right after a fold they may answer the state before it, for as long as the index takes to catch //! up (typically well under a second). The lanes are the local lanes of THAT state. A reader that //! must see its own write reads the ticket's row instead. @@ -68,7 +84,11 @@ //! the selection, at most [`records::READ_CONCURRENCY`] tickets at a time ([`records`]). Folding an //! edge remove reads the add and the remove and rewrites the entry under each end. The graph the //! derived ranking keys read ([`records::with_ranking`]) costs one Query per active ticket and, per -//! inactive neighbour, one `GetItem` plus up to three Queries. No request +//! inactive neighbour, one `GetItem` plus up to three Queries. A hosted `list` +//! ([`records::list_board`]) costs one `typed` Query per listed type and `4n + 1 + p` requests for +//! `n` listed items, as the records read. The `typed` index costs a fold at most one write per item +//! op, and only when the op gains, changes or loses the item's partition — a create, a retype, a +//! delete — in the same write as the item's other index attributes when they move too. No request //! is a Scan: every read names its partition key, the board's through an index. The //! `dynamodb-local` CI job counts the requests of a fold and of a records read, and proves it. //! diff --git a/crates/fold-ddb/src/mem.rs b/crates/fold-ddb/src/mem.rs index 09178aa..155b9b1 100644 --- a/crates/fold-ddb/src/mem.rs +++ b/crates/fold-ddb/src/mem.rs @@ -7,7 +7,7 @@ //! It counts the requests it answers ([`Requests`]), so the cost of a fold — the flag fan-out above //! all — is measured on the same code path the DynamoDB table runs. -use crate::layout::{ACTIVE, DATED, DATED_AT, MAX_ITEM_BYTES, PK, SK}; +use crate::layout::{typed_partition, ACTIVE, DATED, DATED_AT, MAX_ITEM_BYTES, PK, SK, TYPED}; use crate::table::{Dated, Outcome, Row, Table}; use crate::write::{attribute_bytes, beats, Cond, Write}; use nxs_foundation::change::Cell; @@ -176,6 +176,20 @@ impl Table for MemTable { rows.truncate(limit.unwrap_or(usize::MAX)); Ok(rows) } + + async fn query_typed(&self, item_type: &str) -> Result, Infallible> { + self.count(|r| r.index_queries += 1); + let partition = Cell::Text(typed_partition(&self.stream, item_type)); + // The rows are kept by sort key: the index's own order. + Ok(self + .rows + .lock() + .unwrap() + .values() + .filter(|r| r.get(TYPED) == Some(&partition)) + .cloned() + .collect()) + } } /// Run a future that never waits — every [`MemTable`] future — to completion on this thread. diff --git a/crates/fold-ddb/src/records.rs b/crates/fold-ddb/src/records.rs index 4d52e05..6560c95 100644 --- a/crates/fold-ddb/src/records.rs +++ b/crates/fold-ddb/src/records.rs @@ -36,8 +36,17 @@ //! //! One more is read only for a plugin that ranks `next` by derived ranking keys: //! [`with_ranking`] fills the board's [`RankingGraph`] — the edges around the active tickets and -//! their live neighbours — from the adjacency entries (index revision 3, 6j6v.1c5s). Its cost is in -//! its own doc. +//! their live neighbours — from the adjacency entries (index revision 3, 6j6v.1c5s), and the live +//! items of a type whose `[ranking.list]` rank a key reads from the `typed` index (6j6v.xkr7). Its +//! cost is in its own doc. +//! +//! # A hosted `list` +//! +//! [`list_board`] reads what the facade's store-free `list` ranks and projects (6j6v.xkr7): the +//! live, unarchived items of the listed types off the `typed` index, with the joins a `list` +//! record and its filters carry, `next`'s board when a key reads a rank `next` gives, and the +//! graph. The server asks [`read::list_reads`](nexus_flow_facade::read::list_reads) what to read +//! for its query; its cost is in its own doc. //! //! # Two points in time //! @@ -52,27 +61,28 @@ //! # A stream folded by older rules //! //! The entries under a ticket came with index revision 1, the `.cadj#` entries with revision 2 -//! (6j6v.15ed), the self-describing `.adj#` entries with revision 3 (6j6v.1c5s). A stream whose -//! watermark names an older -//! one — or none, folded before revisions were recorded for the index — has no entries, and its -//! records would come back without labels or conversations. [`active_board`] refuses such a stream -//! ([`RecordsError::NotCurrent`]); the fold run refolds it ([`Folder::is_current`]), after which it -//! reads normally. +//! (6j6v.15ed), the self-describing `.adj#` entries and the `typed` index attribute with revision 3 +//! (6j6v.1c5s, 6j6v.xkr7). A stream whose watermark names an older one — or none, folded before +//! revisions were recorded for the index — lacks them, and its records would come back without +//! labels or conversations, its list empty. [`active_board`] and [`list_board`] refuse such a +//! stream ([`RecordsError::NotCurrent`]); the fold run refolds it ([`Folder::is_current`]), after +//! which it reads normally. //! //! [`Folder::is_current`]: crate::fold::Folder::is_current -use crate::board::{int, item_key, Selection, REMOVED}; +use crate::board::{int, is_active, item_key, select, Selection, REMOVED}; use crate::fold::{Folder, INDEX_DOMAIN, INDEX_REVISION}; use crate::layout::{ adjacency_prefix, contributes_adjacency_prefix, label_adjacency_prefix, link_adjacency_prefix, - row_key, + row_key, table_prefix, }; use crate::table::{text, Row, Table}; use futures_util::stream::{self, StreamExt, TryStreamExt}; use nexus_flow_core::graph::Edge; use nexus_flow_core::model::{EdgeKind, ItemRow, LinkWeight}; use nexus_flow_facade::read::{ - ActiveBoard, ActiveTicket, Neighbour, ParentRef, RankingGraph, RankingReads, + ActiveBoard, ActiveTicket, ListBoard, ListItem, ListReads, Neighbour, ParentRef, RankingGraph, + RankingReads, }; use nxs_foundation::change::Cell; use std::collections::{BTreeMap, BTreeSet}; @@ -120,12 +130,7 @@ pub async fn active_board( table: &T, selection: Selection, ) -> Result> { - if let Some(mark) = Folder::watermark(table).await? { - let found = mark.revisions.get(INDEX_DOMAIN).copied(); - if found != Some(INDEX_REVISION) { - return Err(RecordsError::NotCurrent { found }); - } - } + check_current(table).await?; let Selection { items, board, @@ -268,15 +273,19 @@ fn current_parent(id: &str, edges: &BTreeMap) -> Option#` | active ticket | -/// | one `GetItem` of its row | other end of such an edge that is not active | +/// | one Query on the `typed` index | type in `reads.list_types` (6j6v.xkr7) | +/// | one Query on `.adj##` | live, unarchived item of such a type that is not active | +/// | one `GetItem` of its row | other end of such an edge that is not a subject | /// | one Query on `.adj##` (its parent) | such end that is live | -/// | one Query on `.ladj##`, `custom_fields##` | such end, when `reads` asks for labels, custom values | +/// | one Query on `.ladj##`, `custom_fields##` | such end and listed item, when `reads` asks for labels, custom values | /// /// So `n` active tickets with `p` distinct inactive neighbours, `q` of them live, cost `n + q` /// Queries and `p` `GetItem`s — `n + 3q` Queries with labels and custom values — at most /// [`READ_CONCURRENCY`] items at a time, and nothing grows with an item's history: an entry says -/// whether its add is removed. Every edge touching an active ticket is read once, from one end's -/// entries, whatever the plugin's keys — the edge pass the store path runs once per query. +/// whether its add is removed. Every edge touching a subject is read once, from one end's entries, +/// whatever the plugin's keys — the edge pass the store path runs once per query. A `next` key +/// that reads the rank of a type only `[ranking.list]` orders adds, per such type, one index Query +/// and its live items as subjects ([`RankingReads::list_types`]). /// /// [`read::with_ranking`]: nexus_flow_facade::read::with_ranking /// [`read::next_ranking_reads`]: nexus_flow_facade::read::next_ranking_reads @@ -285,28 +294,120 @@ pub async fn with_ranking( mut board: ActiveBoard, reads: &RankingReads, ) -> Result> { - // Every present edge of a kind read that touches an active ticket, once per tag. let ids: Vec = board.tickets.keys().cloned().collect(); - let per_ticket: Vec> = stream::iter(ids) - .map(|id| async move { present_edges(table, &id).await }) + let held: BTreeMap = stream::iter(ids) + .map(|id| async move { + let edges = present_edges(table, &id).await?; + Ok::<_, T::Error>((id, edges)) + }) .buffered(READ_CONCURRENCY) - .try_collect() - .await?; + .try_collect::>() + .await? + .into_iter() + .map(|(id, edges)| { + let parent = board + .tickets + .get(&id) + .and_then(|t| t.item.belongs_to.clone()); + (id, Held { parent, edges }) + }) + .collect(); + board.ranking = Some(ranking_graph(table, reads, held).await?); + Ok(board) +} + +/// A row the board holds itself — an active ticket, a listed item — as the graph reads it: its +/// current parent and the present edges touching it. +struct Held { + parent: Option, + edges: BTreeMap, +} + +/// The live, unarchived items of `item_type` on the `typed` index — the type's `list` scope — +/// asked of the row again, since the index may lag it. In id order. +async fn typed_items(table: &T, item_type: &str) -> Result, T::Error> { + Ok(table + .query_typed(item_type) + .await? + .into_iter() + .filter(|r| { + crate::board::typed(r) == Some(item_type) + && text(r, "archived").is_none() + && text(r, "id").is_some() + }) + .collect()) +} + +/// One live row the graph holds, read in full: its row with its current parent, its labels and +/// custom values when `reads` asks, and its present edges. +async fn neighbour( + table: &T, + id: &str, + row: &Row, + reads: &RankingReads, +) -> Result<(Neighbour, BTreeMap), T::Error> { + let edges = present_edges(table, id).await?; + let mut n = Neighbour::new(item_row(row, current_parent(id, &edges))); + if reads.labels { + n.labels = labels(table, id).await?; + } + if reads.custom { + n.custom = custom(table, id).await?; + } + Ok((n, edges)) +} + +/// The [`RankingGraph`] for `reads` around the rows a board holds itself (`held`, each a subject, +/// with its edges already read) — what [`with_ranking`] and [`list_board`] put on their boards. +async fn ranking_graph( + table: &T, + reads: &RankingReads, + held: BTreeMap, +) -> Result { + // The listed items of `reads.list_types` the board does not hold: subjects too. + let mut listed: BTreeSet = BTreeSet::new(); + let mut scope_rows: Vec<(String, Row)> = Vec::new(); + for ty in &reads.list_types { + for row in typed_items(table, ty).await? { + let id = text(&row, "id").unwrap_or_default().to_string(); + listed.insert(id.clone()); + if !held.contains_key(&id) { + scope_rows.push((id, row)); + } + } + } + let scope: BTreeMap)> = + stream::iter(scope_rows) + .map(|(id, row)| async move { + let read = neighbour(table, &id, &row, reads).await?; + Ok::<_, T::Error>((id, read)) + }) + .buffered(READ_CONCURRENCY) + .try_collect() + .await?; + let subject = |id: &str| held.contains_key(id) || scope.contains_key(id); + + // Every present edge of a kind read that touches a subject, once per tag. let mut touching: BTreeMap = BTreeMap::new(); - for edges in per_ticket { + for edges in held + .values() + .map(|h| &h.edges) + .chain(scope.values().map(|(_, e)| e)) + { touching.extend( edges - .into_iter() - .filter(|(_, e)| reads.kinds.contains(&e.kind)), + .iter() + .filter(|(_, e)| reads.kinds.contains(&e.kind)) + .map(|(tag, e)| (tag.clone(), e.clone())), ); } - // The other ends that are not active: their rows, and for the live ones their current parent - // and the joins asked for. + // The other ends: their rows, and for the live ones their current parent and the joins asked + // for. let others: BTreeSet = touching .values() .flat_map(|e| [&e.from, &e.to]) - .filter(|id| !board.tickets.contains_key(*id)) + .filter(|id| !subject(id)) .cloned() .collect(); let rows: Vec<(String, Option)> = stream::iter(others) @@ -321,16 +422,9 @@ pub async fn with_ranking( .into_iter() .filter_map(|(id, row)| row.map(|r| (id, r))) .filter(|(_, r)| text(r, "deleted") != Some("1")); - let neighbours: BTreeMap = stream::iter(live) + let ends: BTreeMap = stream::iter(live) .map(|(id, row)| async move { - let parent = current_parent(&id, &present_edges(table, &id).await?); - let mut n = Neighbour::new(item_row(&row, parent)); - if reads.labels { - n.labels = labels(table, &id).await?; - } - if reads.custom { - n.custom = custom(table, &id).await?; - } + let (n, _) = neighbour(table, &id, &row, reads).await?; Ok::<_, T::Error>((id, n)) }) .buffered(READ_CONCURRENCY) @@ -339,14 +433,16 @@ pub async fn with_ranking( // The edges between live ends, a `parent` edge only from a child to its current parent. let parent_of = |id: &str| -> Option<&str> { - board - .tickets - .get(id) - .map(|t| &t.item) - .or_else(|| neighbours.get(id).map(|n| &n.item)) - .and_then(|i| i.belongs_to.as_deref()) + match held.get(id) { + Some(h) => h.parent.as_deref(), + None => scope + .get(id) + .map(|(n, _)| n) + .or_else(|| ends.get(id)) + .and_then(|n| n.item.belongs_to.as_deref()), + } }; - let live_end = |id: &str| board.tickets.contains_key(id) || neighbours.contains_key(id); + let live_end = |id: &str| subject(id) || ends.contains_key(id); let edges: BTreeSet = touching .into_values() .filter(|e| live_end(&e.from) && live_end(&e.to)) @@ -358,22 +454,229 @@ pub async fn with_ranking( }) .collect(); // A child whose current parent is elsewhere was read for its parent edge only, which is not in - // the graph: it is no neighbour. - let ends: BTreeSet<&str> = edges + // the graph: it is no neighbour. A listed item always is one. + let edge_ends: BTreeSet<&str> = edges .iter() .flat_map(|e| [e.from.as_str(), e.to.as_str()]) .collect(); - let neighbours = neighbours + let mut neighbours: BTreeMap = ends .into_iter() - .filter(|(id, _)| ends.contains(id.as_str())) + .filter(|(id, _)| edge_ends.contains(id.as_str())) .collect(); + neighbours.extend(scope.into_iter().map(|(id, (n, _))| (id, n))); let mut graph = RankingGraph::new(reads.clone()); graph.edges = edges; graph.neighbours = neighbours; - board.ranking = Some(graph); + graph.listed = listed; + Ok(graph) +} + +/// The [`ListBoard`] for `reads` from the folded tables (6j6v.xkr7) — what [`read::list_board`] +/// reads from a store, item for item — for the facade's store-free `list` +/// ([`read::list_board_page`], [`read::list_board_value`]). A server takes `reads` from +/// [`read::list_reads`] for its query. The stream must be current, as for [`active_board`]: the +/// `typed` index and the self-describing `.adj#` entries came with index revision 3. +/// +/// | read | per | +/// |---------------------------------------------------------|----------------------------------| +/// | one `GetItem` of `.meta#watermark` | board | +/// | one Query on the `typed` index | listed type | +/// | one Query on the partition's `items#` rows | board without a type filter | +/// | one Query each on `.adj#`, `.ladj#`, `custom_fields#`; one `GetItem` of `item_timestamps` | listed item | +/// | one `GetItem` | parent of an open listed item outside the list | +/// | one Query on the `active` index, and per active ticket one Query each on `.ladj#` / `custom_fields#` when the graph reads them | board whose keys read a `next` rank | +/// | [`with_ranking`]'s reads around the listed items and active tickets | board with derived keys | +/// +/// So `n` listed items cost `4n + 1 + p` requests and one Query per listed type, as +/// [`active_board`] costs per active ticket; the `.adj#` Query gives each item's parent, its +/// `contributes_to` targets and its parents for `parent_closed_reason` at once, and the graph reuses +/// it. Archived and deleted items are dropped before any join: `list` returns neither (archived +/// ones are on the index, live as they are, and filtered here as `list` filters them locally). +/// +/// A query without a type filter lists every live item, so it reads the partition's `items#` rows +/// — strongly consistent, every item row of the stream, deleted ones included — instead of an +/// index. The index lists one type; a query that names its types reads only those. +/// +/// [`read::list_board`]: nexus_flow_facade::read::list_board +/// [`read::list_board_page`]: nexus_flow_facade::read::list_board_page +/// [`read::list_board_value`]: nexus_flow_facade::read::list_board_value +/// [`read::list_reads`]: nexus_flow_facade::read::list_reads +pub async fn list_board( + table: &T, + reads: &ListReads, +) -> Result> { + check_current(table).await?; + let rows: Vec = match &reads.types { + Some(types) => { + let mut rows = Vec::new(); + for ty in types { + rows.extend(typed_items(table, ty).await?); + } + rows + } + None => table + .query_prefix(&table_prefix("items")) + .await? + .into_iter() + .filter(|r| { + text(r, "deleted") != Some("1") + && text(r, "archived").is_none() + && text(r, "id").is_some() + }) + .collect(), + }; + let read: Vec<(Row, BTreeMap, ListItem)> = stream::iter(rows) + .map(|row| async move { + let id = text(&row, "id").unwrap_or_default().to_string(); + let edges = present_edges(table, &id).await?; + let (created_at, updated_at) = timestamps(table, &id).await?; + let mut i = ListItem::new(item_row(&row, current_parent(&id, &edges))); + i.labels = labels(table, &id).await?; + i.custom = custom(table, &id).await?; + i.created_at = created_at; + i.updated_at = updated_at; + let targets: BTreeSet = edges + .values() + .filter(|e| e.kind == EdgeKind::ContributesTo && e.from == id) + .map(|e| e.to.clone()) + .collect(); + i.contributes_to = targets.into_iter().collect(); + Ok::<_, T::Error>((row, edges, i)) + }) + .buffered(READ_CONCURRENCY) + .try_collect() + .await?; + + // `parent_closed_reason`: an OPEN item with parents, none of them active, lists them all. A + // parent outside the list is read once, however many children it has. + let parents_of = |id: &str, edges: &BTreeMap| -> BTreeSet { + edges + .values() + .filter(|e| e.kind == EdgeKind::Parent && e.from == id) + .map(|e| e.to.clone()) + .collect() + }; + let listed_rows: BTreeMap<&str, &Row> = read + .iter() + .map(|(row, _, i)| (i.item.id.as_str(), row)) + .collect(); + let outside: BTreeSet = read + .iter() + .filter(|(_, _, i)| i.item.status.as_deref() == Some("open")) + .flat_map(|(_, edges, i)| parents_of(&i.item.id, edges)) + .filter(|p| !listed_rows.contains_key(p.as_str())) + .collect(); + let outside: BTreeMap> = stream::iter(outside) + .map(|pid| async move { + let row = table.get(&item_key(&pid)).await?; + Ok::<_, T::Error>((pid, row)) + }) + .buffered(READ_CONCURRENCY) + .try_collect() + .await?; + let parent_row = |pid: &str| -> Option<&Row> { + listed_rows + .get(pid) + .copied() + .or_else(|| outside.get(pid).and_then(Option::as_ref)) + }; + let parent_closed: Vec)>> = read + .iter() + .map(|(_, edges, i)| { + if i.item.status.as_deref() != Some("open") { + return Vec::new(); + } + let mut closed = Vec::new(); + for pid in parents_of(&i.item.id, edges) { + match parent_row(&pid) { + Some(r) if is_active(r) => return Vec::new(), + Some(r) => closed.push(( + text(r, "id").unwrap_or(&pid).to_string(), + text(r, "closing_comment").map(str::to_string), + )), + None => closed.push((pid, None)), + } + } + closed.sort_by(|a, b| a.0.cmp(&b.0)); + closed + }) + .collect(); + let mut items = Vec::with_capacity(read.len()); + let mut held: BTreeMap = BTreeMap::new(); + for ((_, edges, mut i), closed) in read.into_iter().zip(parent_closed) { + i.parent_closed = closed; + held.insert( + i.item.id.clone(), + Held { + parent: i.item.belongs_to.clone(), + edges, + }, + ); + items.push(i); + } + let mut board = ListBoard::new(reads.types.clone(), items); + + let graph_reads = reads.ranking.clone().unwrap_or_default(); + if reads.next { + let Selection { + items: rows, + board: lanes, + parents, + } = select(table).await?; + let tickets: Vec = stream::iter(rows) + .map(|(id, row)| { + let parent = parents.get(&id).cloned(); + let graph_reads = &graph_reads; + async move { + let mut t = ActiveTicket::new(item_row(&row, parent)); + if graph_reads.labels { + t.labels = labels(table, &id).await?; + } + if graph_reads.custom { + t.custom = custom(table, &id).await?; + } + Ok::<_, T::Error>(t) + } + }) + .buffered(READ_CONCURRENCY) + .try_collect() + .await?; + board.active = Some(ActiveBoard::new(lanes, tickets)); + } + if let Some(r) = &reads.ranking { + if let Some(active) = &board.active { + let missing: Vec<(String, Option)> = active + .tickets + .iter() + .filter(|(id, _)| !held.contains_key(*id)) + .map(|(id, t)| (id.clone(), t.item.belongs_to.clone())) + .collect(); + let more: Vec<(String, Held)> = stream::iter(missing) + .map(|(id, parent)| async move { + let edges = present_edges(table, &id).await?; + Ok::<_, T::Error>((id, Held { parent, edges })) + }) + .buffered(READ_CONCURRENCY) + .try_collect() + .await?; + held.extend(more); + } + board.ranking = Some(ranking_graph(table, r, held).await?); + } Ok(board) } +/// Refuse a stream folded by older index rules than this build reads. +async fn check_current(table: &T) -> Result<(), RecordsError> { + if let Some(mark) = Folder::watermark(table).await? { + let found = mark.revisions.get(INDEX_DOMAIN).copied(); + if found != Some(INDEX_REVISION) { + return Err(RecordsError::NotCurrent { found }); + } + } + Ok(()) +} + /// An `items` row as the store reads it, with the ticket's `belongs_to` (the `present_parent` join). pub(crate) fn item_row(row: &Row, belongs_to: Option) -> ItemRow { let cell = |c: &str| text(row, c).map(str::to_string); diff --git a/crates/fold-ddb/src/table.rs b/crates/fold-ddb/src/table.rs index c382c1e..51cd92b 100644 --- a/crates/fold-ddb/src/table.rs +++ b/crates/fold-ddb/src/table.rs @@ -1,6 +1,6 @@ -//! The five requests a server fold makes of its table, as a trait: [`MemTable`](crate::mem::MemTable) +//! The requests a server fold makes of its table, as a trait: [`MemTable`](crate::mem::MemTable) //! answers them in memory for the tests, `ddb::DynamoDbTable` (feature `dynamodb`) answers them with DynamoDB. -//! None of them is a Scan — every read names its partition, and the two board reads name an index. +//! None of them is a Scan — every read names its partition, and the three index reads name an index. use crate::write::Write; use nxs_foundation::change::Cell; @@ -71,6 +71,13 @@ pub trait Table: Sync { lane: Dated, limit: Option, ) -> impl Future, Self::Error>> + Send; + + /// Every row on the `typed` index under `item_type` (6j6v.xkr7): the live items of that type, + /// archived ones included, in sort-key (id) order. Eventually consistent, as every index read. + fn query_typed( + &self, + item_type: &str, + ) -> impl Future, Self::Error>> + Send; } /// A text cell of `row`, `None` when it is absent or NULL. diff --git a/crates/fold-ddb/tests/common/mod.rs b/crates/fold-ddb/tests/common/mod.rs index eeffa36..8d4274c 100644 --- a/crates/fold-ddb/tests/common/mod.rs +++ b/crates/fold-ddb/tests/common/mod.rs @@ -25,6 +25,10 @@ impl Lcg { } } +/// The item types of a generated board: items are created as one of the first two and retyped +/// among all three, so the `typed` index sees a type gained, lost and changed (6j6v.xkr7). +pub const TYPES: [&str; 3] = ["task", "epic", "note"]; + pub fn id(i: u64) -> String { format!("ab12.{i:04}") } @@ -38,6 +42,11 @@ fn write(s: &mut Store, r: &mut Lcg, n: u64, steps: usize) { step % 24 )); let (a, b) = (id(r.next(n)), id(r.next(n + 1))); + // Now and then a retype (6j6v.xkr7), on the step alone so the random stream — and with it + // every other op of a seed — stays as it was. + if step % 9 == 4 { + s.set_field(&a, "type", Some(TYPES[step % TYPES.len()].into()), "u"); + } match r.next(16) { 0 => s.create_item(&a, "task", "T", "u"), 1 => s.set_field(&a, "status", Some("in_progress".into()), "u"), @@ -149,7 +158,7 @@ pub fn merged(seed: u64) -> Store { let mut a = Store::open_in_memory(1); let mut b = Store::open_in_memory(2); for i in 0..n { - a.create_item(&id(i), "task", "T", "u"); + a.create_item(&id(i), TYPES[(i % 2) as usize], "T", "u"); } b.apply(&a.export()); let steps = 10 + r.next(40) as usize; diff --git a/crates/fold-ddb/tests/dynamodb_local.rs b/crates/fold-ddb/tests/dynamodb_local.rs index 7c49649..d664758 100644 --- a/crates/fold-ddb/tests/dynamodb_local.rs +++ b/crates/fold-ddb/tests/dynamodb_local.rs @@ -1,6 +1,6 @@ //! The DynamoDB adapter against a real DynamoDB Local (6j6v.k7w7): the same fold written into //! DynamoDB and into a [`MemTable`] leaves the same items, the board reads back through the indexes -//! with the same lanes, and every request the client sends is recorded — none is a Scan. +//! with the same lanes and the same `typed` listings (6j6v.xkr7), and every request the client sends is recorded — none is a Scan. //! //! Gated like `nxs-server`'s suite: on the `dynamodb` feature and `NXF_RELAY_DDB_ENDPOINT`, skipped //! locally without one and a hard failure under CI. Each test creates its own table from @@ -16,7 +16,7 @@ use aws_sdk_dynamodb::config::{ use aws_sdk_dynamodb::error::BoxError; use aws_sdk_dynamodb::types::TableStatus; use aws_sdk_dynamodb::Client; -use common::{by_key, delivered, merged, NOW}; +use common::{by_key, delivered, merged, NOW, TYPES}; use nexus_flow_core::model::EdgeKind; use nexus_flow_core::store::Store; use nxs_fold_ddb::ddb::{create_table, DynamoDbTable}; @@ -204,6 +204,20 @@ async fn a_board_folded_into_dynamodb_reads_back_as_from_memory_and_no_request_i assert_eq!(got, want, "{lane:?} limit {limit:?}, seed {seed}"); } } + // The `typed` index (6j6v.xkr7): each type's live items, in id order, through the SDK. + for ty in TYPES { + let ids = |rows: Vec| -> Vec { + rows.iter() + .map(|r| text(r, "id").unwrap().to_string()) + .collect() + }; + let want = ids(mem.query_typed(ty).await.unwrap()); + let got = eventually(&want, || async move { + ids(ddb_ref.query_typed(ty).await.unwrap()) + }) + .await; + assert_eq!(got, want, "typed {ty}, seed {seed}"); + } // Two streams share the table without seeing each other: the second fold's partition // holds only its own rows (checked above), and its reads named its partition. let read = recorder.take(); @@ -309,7 +323,7 @@ async fn a_snapshot_taken_from_dynamodb_starts_another_stream_with_the_same_item // The partition and the index attributes name their stream; everything else is the // same across the two. r.remove("stream_id"); - for attr in ["nxf_active", "nxf_dated"] { + for attr in ["nxf_active", "nxf_dated", "nxf_typed"] { if let Some(nxs_foundation::change::Cell::Text(v)) = r.get_mut(attr) { *v = v .replacen("stream-a", "", 1) diff --git a/crates/fold-ddb/tests/fold_differential.rs b/crates/fold-ddb/tests/fold_differential.rs index ffb765f..824caa8 100644 --- a/crates/fold-ddb/tests/fold_differential.rs +++ b/crates/fold-ddb/tests/fold_differential.rs @@ -6,8 +6,8 @@ //! through the SQLite applier, in the order the replica took it in; and through this crate into a //! [`MemTable`], in a SHUFFLED order with ops delivered twice, as a relay without deduplication //! hands them over. Every row of every view table, every lane of the library and both dated lanes -//! must come out the same, and the incrementally kept indexes must be exactly what a full reindex -//! computes. +//! must come out the same, the `typed` index must list each type's live items (6j6v.xkr7), and the +//! incrementally kept indexes must be exactly what a full reindex computes. use nexus_flow_core::graph; use nexus_flow_core::model::EdgeKind; @@ -15,13 +15,13 @@ use nexus_flow_core::store::Store; use nexus_flow_core::task_reducer::TaskReducer; use nxs_fold_ddb::fold::Folder; use nxs_fold_ddb::mem::{ready, MemTable}; -use nxs_fold_ddb::table::{text, Dated}; +use nxs_fold_ddb::table::{text, Dated, Table}; use nxs_fold_ddb::{board, chunks}; use nxs_foundation::reducer::Reducer; use std::collections::BTreeMap; mod common; -use common::{merged, serve, served_rows, sqlite_rows, NOW}; +use common::{merged, serve, served_rows, sqlite_rows, NOW, TYPES}; fn dated_ids(t: &MemTable, lane: Dated) -> Vec { ready(board::dated(t, lane, None)) @@ -45,6 +45,7 @@ fn the_server_fold_in_any_order_answers_as_the_local_fold() { let folder = Folder::platform().unwrap(); let mut active_seen = 0; let mut chunks_seen = 0; + let mut typed_seen = 0; for seed in 0..120 { let s = merged(seed); let t = serve(&s, seed, &folder); @@ -95,6 +96,24 @@ fn the_server_fold_in_any_order_answers_as_the_local_fold() { "archived lane, {ctx}" ); + // The `typed` index (6j6v.xkr7): under each type exactly its live items, archived ones + // included, in id order. + for ty in TYPES { + let typed: Vec = ready(t.query_typed(ty)) + .unwrap() + .iter() + .map(|r| text(r, "id").unwrap().to_string()) + .collect(); + let want = sql_ids( + &s, + &format!( + "SELECT id FROM items WHERE COALESCE(deleted,'0')<>'1' AND type='{ty}' ORDER BY id" + ), + ); + typed_seen += want.len(); + assert_eq!(typed, want, "typed {ty}, {ctx}"); + } + // The flags kept op by op are exactly what recomputing them from the rows gives. let kept = t.rows(); ready(board::reindex(&t)).unwrap(); @@ -112,6 +131,7 @@ fn the_server_fold_in_any_order_answers_as_the_local_fold() { } assert!(active_seen > 0, "the generated boards have active tickets"); assert!(chunks_seen > 0, "the generated boards supersede chunks"); + assert!(typed_seen > 0, "the generated boards have typed items"); } #[test] diff --git a/crates/fold-ddb/tests/list_differential.rs b/crates/fold-ddb/tests/list_differential.rs new file mode 100644 index 0000000..c44e6fd --- /dev/null +++ b/crates/fold-ddb/tests/list_differential.rs @@ -0,0 +1,441 @@ +//! A server answers a ranked `list` as a replica does (6j6v.xkr7, docs/specs/derived-ranking-keys.md +//! §6 and §9 "differential"), and ranks a `next` whose keys read a `[ranking.list]` rank. +//! +//! Generated boards of four types — goals, projects, actions, persons — written by two replicas +//! apart and then merged: every status, archived and deleted items, retypes, priorities, the goals' +//! `order` custom field, `waiting` labels, and `parent`, `contributes_to`, `mentions` and `dep` +//! edges with removes. Each board is folded into a [`MemTable`] in a shuffled order with +//! duplicates, as the relay hands it over. +//! +//! The plugin ranks with every derived key kind across both sections: persons by a `sum` of +//! mentions, the `list` rank of the goals and the `next` rank of the projects they contribute to; +//! goals by their contributors under `[ranking.list]`; projects under `[ranking.next]` by the +//! `list` rank of their parent goal; actions under `[ranking.next]` by the `list` rank of the +//! persons they mention — the cross-section reads that were refused on the server until the +//! `typed` index — and, in `list`, by the `[ranking.list]` default order with a derived key. +//! +//! Three paths answer each of a set of `list` queries (types, none, container, label, status, +//! sort, limit): +//! +//! 1. the SQL path: `read::list_page` over the store; +//! 2. the Board path: `read::list_board_page` over the store's own [`read::ListBoard`] +//! (`read::list_board` for `read::list_reads`); +//! 3. the fold-ddb path: `read::list_board_page` over the folded tables' (`records::list_board`). +//! +//! All three must give the same rows, order and total, the two boards must be equal item for item +//! and edge for edge, and the JSON must be `Engine::list_query_value`'s. `next` is held to the +//! same three paths. + +use nexus_flow_core::model::{EdgeKind, ItemRow, MergeStrategy}; +use nexus_flow_core::store::Store; +use nexus_flow_facade::plugin::{self, PluginConfig, PluginRegistration}; +use nexus_flow_facade::read::{self, ListQuery, NextFilter, SortKey}; +use nxs_fold_ddb::fold::Folder; +use nxs_fold_ddb::mem::{ready, MemTable}; +use nxs_fold_ddb::{board, records}; +use std::collections::BTreeMap; + +mod common; +use common::{delivered, Lcg, NOW}; + +const ORDERS: &str = r#" +name = "list-differential-fixture" +[description] +en = "x" +de = "y" +[priority] +labels = ["P0", "P1", "P2", "P3", "P4"] +[types] +list = ["goal", "project", "action", "person"] +[vocabulary.status] +open = "open" +in_progress = "in progress" +closed = "closed" +[ranking.next] +types = ["action", "project"] +order = [ { field = "priority", dir = "asc" } ] +[ranking.next.project] +order = [ + { related = "parent", type = "goal", of = "rank", pick = "max", nulls = "first" }, + { related = "contributes_to", type = "goal", of = { field = "order" }, pick = "min", nulls = "last" }, + { count = "children", type = "action", dir = "desc" }, + { field = "priority", dir = "asc" }, +] +[ranking.next.action] +order = [ + { related = "mentions", type = "person", of = "rank", pick = "min", nulls = "last" }, + { related = "parent", type = "project", of = "rank", nulls = "last" }, + { related = "dep", of = { field = "status", precedence = ["closed", "in_progress", "open"] }, pick = "min" }, + { field = "priority", dir = "asc" }, +] +[ranking.list] +types = ["person", "goal", "project"] +order = [ + { count = "mentioned_by", dir = "desc" }, + { field = "priority", dir = "asc" }, +] +[ranking.list.goal] +order = [ + { count = "contributors", type = "project", dir = "desc" }, + { field = "priority", dir = "asc" }, +] +[ranking.list.person] +order = [ + { sum = [ { count = "mentioned_by", type = "project" }, + { count = "mentioned_by", type = "action", label = "waiting", status = ["open", "in_progress"] } ], dir = "desc" }, + { related = "contributes_to", type = "goal", of = "rank", pick = "min", nulls = "last" }, + { related = "contributes_to", type = "project", of = "rank", pick = "min", nulls = "last" }, + { count = "dependents", status = ["closed"], dir = "desc" }, + { field = "priority", dir = "asc" }, +] +[presentation.list] +columns = ["id"] +[fields.order] +type = "text" +on = ["goal"] +"#; + +inventory::submit!(PluginRegistration { + name: "list-differential-fixture", + toml: ORDERS, + order: 141, +}); + +fn cfg() -> PluginConfig { + plugin::load("list-differential-fixture").expect("the fixture is a valid plugin") +} + +const TYPES: [&str; 4] = ["goal", "project", "action", "person"]; + +/// The items of a generated board: three goals, four projects, six actions, five persons. +fn items() -> Vec<(String, &'static str)> { + let mut out = Vec::new(); + for (ty, letter, n) in [ + ("goal", 'g', 3), + ("project", 'p', 4), + ("action", 'a', 6), + ("person", 'u', 5), + ] { + for i in 0..n { + out.push((format!("ab12.{letter}{i:03}"), ty)); + } + } + out +} + +/// One replica's share of the writing. +fn write(s: &mut Store, r: &mut Lcg, added: &mut Vec<(String, String, EdgeKind)>, steps: usize) { + let all = items(); + let pick = |r: &mut Lcg| all[r.next(all.len() as u64) as usize].0.clone(); + for step in 0..steps { + s.set_wall_clock(&format!( + "2026-10-0{}T{:02}:00:00Z", + 1 + r.next(7), + step % 24 + )); + let (x, y) = (pick(r), pick(r)); + match r.next(18) { + 0 => s.set_field(&x, "status", Some("in_progress".into()), "u"), + 1 | 2 => s.set_field(&x, "status", Some("closed".into()), "u"), + 3 => s.set_field(&x, "status", Some("open".into()), "u"), + 4 => s.set_field( + &x, + "archived", + Some(format!("2026-10-0{}", 1 + r.next(7))), + "u", + ), + 5 => s.set_field(&x, "archived", None, "u"), + 6 if r.next(3) == 0 => s.delete_item(&x, "u"), + 7 => s.set_field(&x, "priority", Some(r.next(5).to_string()), "u"), + 8..=12 => { + let kind = match r.next(10) { + 0..=2 => EdgeKind::Parent, + 3..=5 => EdgeKind::ContributesTo, + 6..=8 => EdgeKind::Mentions, + _ => EdgeKind::Dep, + }; + s.add_edge(&x, &y, kind, "u"); + added.push((x, y, kind)); + } + 13 if !added.is_empty() => { + let (from, to, kind) = added.swap_remove(r.next(added.len() as u64) as usize); + s.remove_edge(&from, &to, kind, "u"); + } + 14 => { + if r.next(3) == 0 { + s.remove_label(&x, "waiting", "u"); + } else { + s.add_label(&x, "waiting", "u"); + } + } + 15 if r.next(3) == 0 => { + s.set_field(&x, "type", Some(TYPES[r.next(4) as usize].into()), "u") + } + _ => s.set_custom_field_merge( + &x, + "order", + Some(r.next(6).to_string()), + "u", + MergeStrategy::Lww, + ), + } + } +} + +/// A merged board: two replicas that wrote apart, then exchanged everything. +fn gen_board(seed: u64) -> Store { + let mut r = Lcg(seed ^ 0x115d); + let mut a = Store::open_in_memory(1); + for (id, ty) in items() { + a.create_item(&id, ty, "T", "u"); + a.set_field(&id, "priority", Some(r.next(5).to_string()), "u"); + if ty == "goal" && r.next(4) != 0 { + a.set_custom_field_merge( + &id, + "order", + Some(r.next(6).to_string()), + "u", + MergeStrategy::Lww, + ); + } + } + let mut b = Store::open_in_memory(2); + b.apply(&a.export()); + let steps = 30 + r.next(40) as usize; + write(&mut a, &mut r, &mut Vec::new(), steps); + write(&mut b, &mut r, &mut Vec::new(), steps); + let from_b = b.export(); + a.apply(&from_b); + a +} + +fn ids(items: &[ItemRow]) -> Vec { + items.iter().map(|i| i.id.clone()).collect() +} + +/// The `list` queries each board is asked: a type, several, none; a container, a label, a status; +/// the default order; a cut. +fn queries(container: &str) -> Vec { + let ranked = || ListQuery::new().sort(SortKey::Rank).now(NOW); + let types = |tys: &[&str]| tys.iter().fold(NextFilter::new(), |f, t| f.with_type(*t)); + vec![ + ranked().filter(types(&["person"])), + ranked().filter(types(&["goal"])), + ranked().filter(types(&["project", "person"])), + ranked(), + ranked().filter(types(&["action"]).with_container(container)), + ranked().filter(NextFilter::new().with_label("waiting")), + ranked().status("closed").filter(types(&["person"])), + ListQuery::new().filter(types(&["person"])), + ranked().filter(types(&["person"])).limit(3), + ] +} + +/// What the generated boards held, so the comparison is known to have compared something worth +/// comparing. +#[derive(Default)] +struct Seen(BTreeMap<&'static str, usize>); + +impl Seen { + fn add(&mut self, what: &'static str, n: usize) { + *self.0.entry(what).or_default() += n; + } +} + +#[test] +fn the_server_answers_a_ranked_list_as_the_replica_does() { + let folder = Folder::platform().unwrap(); + let cfg = cfg(); + let mut seen = Seen::default(); + for seed in 0..40 { + let s = gen_board(seed); + let ctx = format!("seed {seed}"); + let t = MemTable::new("stream-1"); + ready(folder.fold_batch(&t, &delivered(&s, seed))).unwrap(); + + for (n, q) in queries(&format!("ab12.p{:03}", seed % 4)) + .iter() + .enumerate() + { + let ctx = format!("{ctx}, query {n}"); + let reads = read::list_reads(&cfg, q); + let local = read::list_board(&s, &reads).unwrap(); + let server = ready(records::list_board(&t, &reads)).unwrap(); + assert_eq!(server.items, local.items, "listed items, {ctx}"); + assert_eq!(server.ranking, local.ranking, "the ranking graph, {ctx}"); + assert_eq!( + server.active.as_ref().map(|a| &a.tickets), + local.active.as_ref().map(|a| &a.tickets), + "next's board, {ctx}" + ); + + // 1 = 2 = 3: the SQL path, the Board path, the fold-ddb path. + let want = read::list_page(&cfg, &s, q).unwrap(); + let board_path = read::list_board_page(&cfg, &local, q).unwrap(); + let served = read::list_board_page(&cfg, &server, q).unwrap(); + assert_eq!( + ids(&board_path.items), + ids(&want.items), + "SQL vs Board, {ctx}" + ); + assert_eq!( + ids(&served.items), + ids(&want.items), + "SQL vs fold-ddb, {ctx}" + ); + assert_eq!(served.total, want.total, "total, {ctx}"); + let records = read::list_to_value_with_custom(&cfg, &s, &want.items).unwrap(); + assert_eq!( + read::list_board_value(&cfg, &server, q).unwrap(), + read::next_page_envelope(records, &want, false), + "list JSON, {ctx}" + ); + + seen.add("listed rows", want.items.len()); + if let Some(graph) = &server.ranking { + seen.add("graph listed items", graph.listed.len()); + seen.add("graph edges", graph.edges.len()); + } + seen.add( + "boards with next's board", + usize::from(server.active.is_some()), + ); + seen.add( + "listed items with a closed-parent reason", + server + .items + .values() + .filter(|i| !i.parent_closed.is_empty()) + .count(), + ); + seen.add( + "closed persons listed", + want.items + .iter() + .filter(|i| { + i.item_type.as_deref() == Some("person") + && i.status.as_deref() == Some("closed") + }) + .count(), + ); + } + + // `next` whose keys read the `list` rank of persons and goals (spec §12 point 1). + let reads = read::next_ranking_reads(&cfg) + .unwrap() + .expect("derived next keys"); + let selection = ready(board::select(&t)).unwrap(); + let plain = ready(records::active_board(&t, selection)).unwrap(); + let plain = ready(records::with_contributes_to(&t, plain)).unwrap(); + let server = ready(records::with_ranking(&t, plain, &reads)).unwrap(); + let local = read::with_ranking(&s, read::active_board(&s).unwrap(), &reads).unwrap(); + assert_eq!(server.ranking, local.ranking, "next's graph, {ctx}"); + let want = read::next(&cfg, &s, NOW, None).unwrap(); + assert_eq!( + ids(&read::next_active(&cfg, &local, NOW).unwrap()), + ids(&want), + "next: SQL vs Board, {ctx}" + ); + assert_eq!( + ids(&read::next_active(&cfg, &server, NOW).unwrap()), + ids(&want), + "next: SQL vs fold-ddb, {ctx}" + ); + seen.add("next rows", want.len()); + seen.add( + "archived items", + s.list_items() + .unwrap() + .iter() + .filter(|i| i.archived.is_some() && i.deleted.as_deref() != Some("1")) + .count(), + ); + } + for what in [ + "listed rows", + "graph listed items", + "graph edges", + "boards with next's board", + "listed items with a closed-parent reason", + "closed persons listed", + "next rows", + "archived items", + ] { + assert!( + seen.0.get(what).copied().unwrap_or(0) > 0, + "the generated boards have {what}: {:?}", + seen.0 + ); + } +} + +#[test] +fn a_stream_not_folded_by_this_index_revision_is_refused_for_a_list() { + // The `typed` index and the self-describing adjacency entries come with index revision 3: a + // stream folded before has neither, and its list would come back empty — refused instead. + let folder = Folder::platform().unwrap(); + let cfg = cfg(); + let s = gen_board(5); + let t = MemTable::new("stream-1"); + ready(folder.fold_batch(&t, &delivered(&s, 5))).unwrap(); + use nxs_fold_ddb::fold::INDEX_DOMAIN; + use nxs_fold_ddb::layout::WATERMARK_KEY; + use nxs_fold_ddb::table::Table; + use nxs_fold_ddb::write::{Cond, Write}; + use nxs_foundation::change::Cell; + ready(t.write(&Write { + sk: WATERMARK_KEY.into(), + set: vec![(format!("revision_{INDEX_DOMAIN}"), Cell::Int(2))], + remove: Vec::new(), + condition: Some(Cond::RowPresent), + })) + .unwrap(); + let q = &queries("ab12.p000")[0]; + match ready(records::list_board(&t, &read::list_reads(&cfg, q))) { + Err(records::RecordsError::NotCurrent { found: Some(2) }) => {} + other => panic!("a revision-2 stream must be refused, got {other:?}"), + } +} + +#[test] +fn reading_a_typed_list_costs_one_index_query_and_four_requests_per_item() { + // §9 "bounded cost": the persons of a board are one Query on the `typed` index, and each costs + // its adjacency, labels, custom values and timestamps — whatever else the board holds, which + // this list reads nothing of. A type filter reads only its own type. + let folder = Folder::platform().unwrap(); + let cfg = cfg(); + let mut s = Store::open_in_memory(1); + let n = 8u64; + for i in 0..n { + s.create_item(&format!("ab12.u{i:03}"), "person", "P", "u"); + } + for i in 0..30 { + s.create_item(&format!("ab12.a{i:03}"), "action", "A", "u"); + } + s.create_item("ab12.uzzz", "person", "Archived", "u"); + s.set_field("ab12.uzzz", "archived", Some("2026-10-01".into()), "u"); + let t = MemTable::new("stream-1"); + ready(folder.fold_batch(&t, &delivered(&s, 1))).unwrap(); + let q = ListQuery::new().filter(NextFilter::new().with_type("person")); + let reads = read::list_reads(&cfg, &q); + assert_eq!(reads.ranking, None); + assert!(!reads.next); + let before = t.requests(); + let server = ready(records::list_board(&t, &reads)).unwrap(); + let after = t.requests(); + assert_eq!(after.index_queries - before.index_queries, 1); + assert_eq!(after.queries - before.queries, 3 * n); + // The watermark, and each person's timestamps; no parents. + assert_eq!(after.gets - before.gets, 1 + n); + assert_eq!(after.writes, before.writes); + assert_eq!( + server.items.len(), + n as usize, + "the archived person is not listed" + ); + let page = read::list_board_page(&cfg, &server, &q).unwrap(); + assert_eq!( + ids(&page.items), + ids(&read::list_page(&cfg, &s, &q).unwrap().items) + ); +} diff --git a/crates/fold-ddb/tests/robustness.rs b/crates/fold-ddb/tests/robustness.rs index 414a3f3..815e3b1 100644 --- a/crates/fold-ddb/tests/robustness.rs +++ b/crates/fold-ddb/tests/robustness.rs @@ -1,6 +1,7 @@ //! What a server fold does when things go wrong (review of PR #24): a run that dies at any write //! and folds its batch again, an op the table cannot hold, a clear that dies half way, a release -//! with other fold revisions, a lane cut by a limit, and a snapshot that does not fit. +//! with other fold revisions, a lane cut by a limit, a snapshot that does not fit — and the `typed` +//! index kept through retypes, archives and deletes (6j6v.xkr7). mod common; @@ -94,6 +95,10 @@ impl Table for Failing<'_> { async fn query_dated(&self, lane: Dated, limit: Option) -> Result, Injected> { Ok(ok(self.inner.query_dated(lane, limit).await)) } + + async fn query_typed(&self, item_type: &str) -> Result, Injected> { + Ok(ok(self.inner.query_typed(item_type).await)) + } } /// Fold `first` cleanly, then `second` with a run that dies at each write in turn and folds @@ -214,6 +219,107 @@ fn an_op_the_table_cannot_hold_is_refused_and_the_stream_folds_on() { assert_eq!(lanes.ready, vec!["ab12.0001", "ab12.0002"]); } +#[test] +fn a_type_too_long_for_the_typed_index_is_refused_and_the_item_folds_on_without_it() { + // 6j6v.xkr7: the `typed` index partition carries the type; DynamoDB refuses an index key past + // 2,048 bytes, and a refused write would fail every run at the same op. So the op that sets + // such a type is refused before it writes, as an op with a key too long is, and the item keeps + // the type it had. + let folder = Folder::platform().unwrap(); + let mut s = Store::open_in_memory(1); + s.create_item("ab12.0001", "task", "T", "u"); + s.set_field("ab12.0001", "type", Some("t".repeat(2100)), "u"); + s.create_item("ab12.0002", &"n".repeat(2030), "Just fits", "u"); + let t = MemTable::new("stream-1"); + let refused = ready(folder.fold_batch(&t, &numbered(&s, 0))).unwrap(); + let reasons: Vec<&str> = refused.iter().map(|r| r.reason.as_str()).collect(); + assert_eq!(refused.len(), 1, "{reasons:#?}"); + assert!( + reasons[0].contains("`typed` index partition"), + "{reasons:#?}" + ); + let ids = |ty: &str| -> Vec { + ready(t.query_typed(ty)) + .unwrap() + .iter() + .map(|r| text(r, "id").unwrap().to_string()) + .collect() + }; + assert_eq!(ids("task"), ["ab12.0001"]); + assert_eq!(ids(&"n".repeat(2030)), ["ab12.0002"]); +} + +#[test] +fn retyping_archiving_and_deleting_move_an_item_on_the_typed_index() { + // 6j6v.xkr7: an item is listed under its type while it is live — archived or not — moves with a + // retype, and leaves when it is deleted. A type with `#` in it gets its own partition. + let folder = Folder::platform().unwrap(); + let mut s = Store::open_in_memory(1); + s.create_item("ab12.0001", "person", "Ada", "u"); + s.create_item("ab12.0002", "person", "Bob", "u"); + s.create_item("ab12.0003", "a#b", "Odd", "u"); + let t = MemTable::new("stream-1"); + let mut seen = 0; + let mut step = |s: &Store| { + let ops = numbered(s, seen); + seen += ops.len(); + ready(folder.fold_batch(&t, &ops)).unwrap(); + ["person", "project", "a#b", "a"].map(|ty| { + ready(t.query_typed(ty)) + .unwrap() + .iter() + .map(|r| text(r, "id").unwrap().to_string()) + .collect::>() + }) + }; + let v = |ids: &[&str]| ids.iter().map(|s| s.to_string()).collect::>(); + let none = v(&[]); + + assert_eq!( + step(&s), + [ + v(&["ab12.0001", "ab12.0002"]), + none.clone(), + v(&["ab12.0003"]), + none.clone() + ] + ); + s.set_field("ab12.0002", "archived", Some("2026-10-08".into()), "u"); + s.set_field("ab12.0001", "status", Some("closed".into()), "u"); + assert_eq!( + step(&s), + [ + v(&["ab12.0001", "ab12.0002"]), + none.clone(), + v(&["ab12.0003"]), + none.clone() + ], + "closed and archived items stay listed" + ); + s.set_field("ab12.0001", "type", Some("project".into()), "u"); + assert_eq!( + step(&s), + [ + v(&["ab12.0002"]), + v(&["ab12.0001"]), + v(&["ab12.0003"]), + none.clone() + ] + ); + s.delete_item("ab12.0002", "u"); + s.delete_item("ab12.0003", "u"); + assert_eq!( + step(&s), + [none.clone(), v(&["ab12.0001"]), none.clone(), none.clone()] + ); + // The attribute is on the item rows only, never on an adjacency entry or another row. + assert!(t + .rows() + .iter() + .filter(|(_, r)| r.contains_key("nxf_typed")) + .all(|(sk, _)| sk.starts_with("items#"))); +} + #[test] fn a_clear_that_dies_keeps_the_watermark_and_a_release_with_other_rules_refolds_from_zero() { let folder = Folder::platform().unwrap(); diff --git a/docs/specs/E4-sync-protocol-server.md b/docs/specs/E4-sync-protocol-server.md index a63c679..9dc3d5c 100644 --- a/docs/specs/E4-sync-protocol-server.md +++ b/docs/specs/E4-sync-protocol-server.md @@ -278,6 +278,13 @@ relay — does it with `nxs-fold-ddb`: the reducers a local replica registers, t - **GSI `dated`** (`nxf_dated` / `nxf_dated_at`): sparse — `#closed` by `closed_at`, `#archived` by `archived`, newest first; the sort key is `U+001F`, so tickets of one instant keep one order and a `limit` cuts the lane at the same place every time. +- **GSI `typed`** (`nxf_typed` / `sk`, both S, projecting all; added 2026-10-11, `6j6v.xkr7`): + sparse — `#` (the type escaped as a key value: `%` → `%25`, `#` → `%23`) on every + live item row, closed and archived ones included, removed on a delete. One Query lists a type's + live items, the `list` scope a ranked hosted `list` and the derived ranking keys rank over + (`records::list_board`, `docs/specs/derived-ranking-keys.md` §6). The deploying stack adds the + GSI first: a fold against a table without it still writes the attribute harmlessly, and only + the read needs the index. - **The active tickets' records** (added 2026-10-10, `6j6v.t1ym`). The LANES need only the `active` Query above. The RECORDS a lane is shown with need more: a lane's JSON carries joins kept in other tables — labels, custom values, timestamps, bearing thread links, the parent's diff --git a/docs/specs/derived-ranking-keys.md b/docs/specs/derived-ranking-keys.md index cf37db7..3682d4f 100644 --- a/docs/specs/derived-ranking-keys.md +++ b/docs/specs/derived-ranking-keys.md @@ -3,8 +3,9 @@ **Status: decided by the owner on 2026-10-09** (§10). The cuts that build it are listed in §11. **Built locally (x0wf, §12):** the three key kinds, their load-time rules and the type chain over the SQL path, in `next` and `list`. **Built on the server for `next` (1c5s, §6):** the same keys -over the hosted board, with the rule the owner decided on 2026-10-11 (§10.5). The `typed` index and -a ranked hosted `list` (xkr7) are open. +over the hosted board, with the rule the owner decided on 2026-10-11 (§10.5). **Built on the server +for `list` (xkr7, §6):** the `typed` index, a ranked hosted `list`, and a hosted `next` whose keys +read a `[ranking.list]` rank. Fourth cut of `6j6v.kfs7` (cut with the owner on 2026-10-09). The three cuts before it add the plumbing this one ranks with: @@ -164,14 +165,18 @@ derived key pays nothing. adjacency; per neighbour that is not active one `GetItem` of its row and, when it is live, one Query on its own adjacency for its current parent, plus one Query each for its labels and custom values when a key reads them. -- **Refused on the server until xkr7:** a `[ranking.next]` order that reads — directly or down its - type chain — the rank of a type only `[ranking.list]` orders (§12, first point). That rank is - computed over the type's live items, which the active board does not hold. The store path ranks - such a plugin. Every other derived `next` key ranks on the server. +- **A `[ranking.list]` rank in a `next` key** (refused on the server in 1c5s, ranked since xkr7): a + `[ranking.next]` order that reads — directly or down its type chain — the rank of a type only + `[ranking.list]` orders (§12, first point). That rank is computed over the type's live, + unarchived items, which the active board does not hold; `read::next_ranking_reads` names such a + type in `RankingReads::list_types`, and the server reads its items off the `typed` index into the + graph (`RankingGraph::listed`), with the edges around them. - **What xkr7 builds on.** The self-describing adjacency entries are written for EVERY edge between - ANY items, so xkr7's "widened adjacency" needs no new entries for the edges: a `list`-ranked item - reads its edges with the same Query. What xkr7 adds is the `typed` index (the `list` scope of a - type), the read that ranks a type's live items from it, and lifting the refusal above. + ANY items, so xkr7's "widened adjacency" needed no new entries for the edges: a `list`-ranked item + reads its edges with the same Query (verified in xkr7: its `.adj#` entries give its current + parent, its `contributes_to` targets, its parents for `parent_closed_reason` and every edge a key + follows or counts). What xkr7 adds is the `typed` index (the `list` scope of a type), the read + that ranks a type's live items from it, and lifting the refusal above. **`list` needs a second index (decided §10.4).** Objects that `list` ranks, such as a person, are long-lived and have no "active" status. If they did, they would rank through `next`, and `list` @@ -191,8 +196,7 @@ gains a second sparse index, keyed by type: (`.adj#`) were always written for every edge add, under both ends; only the `active` INDEX flag is limited to `dep`/`parent` edges with an active end. Since index revision 3 the entries say what their edge is and whether it is removed (`next` above), so a `list`-ranked item reads its edges - with one Query on its own prefix, at no extra cost per edge write. What stays open for xkr7 is - the read of a type's live items through `typed` and their neighbours' rows. + with one Query on its own prefix, at no extra cost per edge write. - **Archived items.** They are in the index, because they are live. `list` filters them at read, as it does locally. - **It changes the table contract.** The deploying stack (the web package's CDK) must add the GSI @@ -200,6 +204,32 @@ gains a second sparse index, keyed by type: still writes the attribute, which costs nothing. Only the `list` read needs the index. The rollout order is therefore: the stack first, then the read. +**`list` as built (xkr7).** +- **The key.** `nxf_typed` is `#` with the type escaped as a key value is (`%` → + `%25`, `#` → `%23`), so `a` + `b#c` and `a#b` + `c` cannot share a partition; the sort key is the + table's `sk`, so a Query lists a type's items in id order. An item without a type is on no + partition. A type whose partition would pass the 2,048 bytes DynamoDB allows an index key is + refused at fold time, as an op with a sort key too long is. +- **Index revision.** The attribute ships in revision 3 with the self-describing adjacency entries + (1c5s and xkr7 reach the same beta), so revision 3 still makes every older stream not current + and one refold sets both; no revision 4. +- **Where it lives.** The facade's `read::list_reads(cfg, query)` names what a hosted `list` reads + (`ListReads`: the listed types — the query's type filter, else every type —, whether `next`'s + board is needed, and the graph); `nxs_fold_ddb::records::list_board` reads it into a `ListBoard` + (the listed items with the joins a `list` record and its filters carry, `next`'s board, the + `RankingGraph`); `read::list_board_page` and `read::list_board_value` filter, rank and project it + by the code `list_page` runs, and `read::list_board` reads the same board from a store, the + parity oracle. +- **Without a type filter** the server lists every live item: it reads the partition's `items#` + rows (strongly consistent, one Query, no index), since the index lists one type at a time. +- **Paging.** As for the hosted `next`: the store path's page cache is machine-local and a + server's paging is an open decision (15ed), so the store-free `list` is not paged — `limit` cuts + it with the total disclosed, `paginate` and a token are refused with `validation`, and a server + cuts the list itself. +- **`now`.** A `list` key that reads a rank `next` gives takes `next`'s candidates at the query's + `now`, as locally (§12); without one the read answers `validation` where a ranked row needs it, + and `next`'s board is not read for it. + A differential test holds the SQL path and the server path to the same order on generated boards, for `next` and for `list`, as for the lanes today. @@ -259,7 +289,10 @@ the plugin declares (`plugin-custom-fields.md`), as decided in §10.2. - **Differential:** SQL path against the server path on generated boards with derived keys, as `fold_differential` does for the lanes. Built for `next` in 1c5s (`crates/fold-ddb/tests/derived_differential.rs`): the SQL path, the store-free path over the - store's own board and graph, and the same over the folded tables give one order. + store's own board and graph, and the same over the folded tables give one order. Built for + `list` in xkr7 (`crates/fold-ddb/tests/list_differential.rs`): the same three paths on + generated boards of four types with retypes, every key kind and cross-section ranks both ways, + over a set of queries (types, none, container, label, status, sort, limit), and the JSON. - **Determinism:** shuffled edge order gives the same result, and ties fall to the id. - **Bounded cost:** a ratio guard in the style of `xn8s`. A query with derived keys runs the edge pass once, not once per candidate. @@ -288,8 +321,8 @@ the plugin declares (`plugin-custom-fields.md`), as decided in §10.2. adjacency entries (index revision 3) and the server reads the neighbours that are not active. Narrowing the local rule to active neighbours was declined, as was refusing on the server every key whose value can depend on a neighbour that is not active (it would refuse §8's first order). - A `[ranking.next]` order that reads the rank of a type only `[ranking.list]` orders is refused on - the server until the `typed` index (xkr7) gives it that type's scope. + A `[ranking.next]` order that reads the rank of a type only `[ranking.list]` orders was refused on + the server until the `typed` index (xkr7) gave it that type's scope; it ranks there since. ## 11. Cuts @@ -299,8 +332,11 @@ the plugin declares (`plugin-custom-fields.md`), as decided in §10.2. lanes' board, the self-describing adjacency entries (index revision 3), and the differential test (SQL path = Board path = fold-ddb path on generated boards with closed, archived and deleted neighbours, mentions and inbound `contributes_to`). -- **The `typed` index and a ranked hosted `list`.** The GSI, the widened adjacency, the read, and - the table-contract note for the deploying stack, coordinated with app-foundations. +- **The `typed` index and a ranked hosted `list`** (xkr7, built). The GSI, the read + (`ListBoard`, `records::list_board`, `read::list_board_page`), the `[ranking.list]` rank in a + hosted `next`, the differential test, and the table-contract note for the deploying stack + (`nxs-fold-ddb` crate doc, E4 §5.1), coordinated with app-foundations. The widened adjacency + turned out to be 1c5s's self-describing entries; nothing more was needed. - **Shared with `6j6v.q29p`.** The reverse mention lookup and `count = "mentioned_by"` use one implementation. @@ -334,7 +370,9 @@ reading of the text, so a later decision can loosen it without breaking a declar takes them. - **`nulls` on a `count` or `sum`** is refused: a count is never null (§3.2). - **The store-free lanes** (`read::next_active`) rank a `[ranking.next]` with derived keys from - the board's `RankingGraph` since 1c5s (§6), with the live-neighbour rule above; they refuse only - an order that reads the rank of a type only `[ranking.list]` orders, until xkr7. + the board's `RankingGraph` since 1c5s (§6), with the live-neighbour rule above; since xkr7 an + order that reads the rank of a type only `[ranking.list]` orders too, from the graph's listed + items. The store-free `list` (`read::list_board_page`, xkr7) ranks a `[ranking.list]` the same + way. - **TOML.** An inline table may not span lines outside its arrays (TOML 1.0), so a `sum`'s `dir` stands on the line its array closes on. §8 above breaks that line for reading only.