diff --git a/Cargo.lock b/Cargo.lock index b695c61..e10c7f3 100644 --- a/Cargo.lock +++ b/Cargo.lock @@ -2442,6 +2442,7 @@ version = "0.209.0" dependencies = [ "aws-sdk-dynamodb", "futures-util", + "inventory", "nexus-chat", "nexus-flow-core", "nexus-flow-facade", diff --git a/changes/1c5s-server-derived-keys.md b/changes/1c5s-server-derived-keys.md new file mode 100644 index 0000000..58aed9e --- /dev/null +++ b/changes/1c5s-server-derived-keys.md @@ -0,0 +1,8 @@ +--- +type: added +facade: changed +--- +[en] +A hosted board now ranks `next` by derived ranking keys (`related`, `count`, `sum`) exactly as a local replica does. The store-free `next` (`read::next_active`, `next_active_value`, `next_active_filtered`) ranks a plugin with such keys from a new `ActiveBoard::ranking` graph: the edges around the active tickets and the live neighbours they reach, closed and archived ones included, deleted ones not. A server fills it with `nxs_fold_ddb::records::with_ranking` for what `read::next_ranking_reads(cfg)` names; `read::with_ranking` reads the same graph from a store. Without the graph, `next_active` still refuses such a plugin. Everything is additive: `ActiveBoard` gains a field (it is `#[non_exhaustive]`), and `RankingReads`, `RankingGraph`, `Neighbour`, `next_ranking_reads` and `with_ranking` are new. For a server's operators: the fold's index revision goes from 2 to 3, so every hosted stream is not current and is folded again on its next run; until then its records read answers `NotCurrent`. The `.adj#` adjacency entries gain attributes; the one table-contract change of this release, the `typed` index, is described with the hosted `list`. +[de] +Ein gehostetes Brett rankt `next` jetzt nach abgeleiteten Ranking-Schlüsseln (`related`, `count`, `sum`) genau so wie ein lokales Replikat. Das speicherfreie `next` (`read::next_active`, `next_active_value`, `next_active_filtered`) rankt ein Plugin mit solchen Schlüsseln aus einem neuen Graphen `ActiveBoard::ranking`: den Kanten um die aktiven Tickets und den lebenden Nachbarn, die sie erreichen, geschlossene und archivierte eingeschlossen, gelöschte nicht. Ein Server füllt ihn mit `nxs_fold_ddb::records::with_ranking` für das, was `read::next_ranking_reads(cfg)` nennt; `read::with_ranking` liest denselben Graphen aus einem Speicher. Ohne den Graphen lehnt `next_active` ein solches Plugin weiterhin ab. Alles ist additiv: `ActiveBoard` bekommt ein Feld (es ist `#[non_exhaustive]`), und `RankingReads`, `RankingGraph`, `Neighbour`, `next_ranking_reads` und `with_ranking` sind neu. Für den Betrieb eines Servers: Die Index-Revision des Folds steigt von 2 auf 3, daher ist jeder gehostete Stream nicht aktuell und wird bei seinem nächsten Lauf neu gefaltet; bis dahin antwortet sein Records-Lesen mit `NotCurrent`. Die `.adj#`-Adjazenzeinträge bekommen Attribute; die eine Änderung des Tabellenvertrags in diesem Release, der Index `typed`, ist bei der gehosteten `list` beschrieben. diff --git a/changes/xkr7-typed-index.md b/changes/xkr7-typed-index.md new file mode 100644 index 0000000..31fb4a1 --- /dev/null +++ b/changes/xkr7-typed-index.md @@ -0,0 +1,8 @@ +--- +type: added +facade: changed +--- +[en] +A hosted board now answers `list` ranked by the plugin's `[ranking.list]`, derived ranking keys included, exactly as a local replica does. The facade names what to read for a query (`read::list_reads`, a new `ListReads`), a server reads it into a new `ListBoard` with `nxs_fold_ddb::records::list_board`, and `read::list_board_page` / `read::list_board_value` filter (type, container, label, status), rank and project it with the code `list` runs locally; `read::list_board` reads the same board from a store. The hosted list is not paged: `limit` cuts it with the total disclosed, and `paginate` or a token is refused, as for the hosted `next`. A hosted `next` whose keys read the rank `[ranking.list]` gives a type now ranks too instead of being refused: `RankingReads` gains `list_types` and `RankingGraph` gains `listed` (both `#[non_exhaustive]`, so additive). For a server's operators, the table contract changes: a third sparse index, GSI `typed` — partition key `nxf_typed` (string, `#`), sort key `sk` (string), projection ALL — set by the fold on every live item row (closed and archived included, removed on delete). Add the index to the stack first, then deploy the read; a fold against a table without the index writes the attribute harmlessly. It ships in fold index revision 3 together with the self-describing adjacency entries, so one refold covers both. +[de] +Ein gehostetes Brett beantwortet `list` jetzt nach dem `[ranking.list]` des Plugins gerankt, abgeleitete Ranking-Schlüssel eingeschlossen, genau so wie ein lokales Replikat. Die Fassade nennt, was für eine Anfrage zu lesen ist (`read::list_reads`, ein neues `ListReads`), ein Server liest es mit `nxs_fold_ddb::records::list_board` in ein neues `ListBoard`, und `read::list_board_page` / `read::list_board_value` filtern (Typ, Container, Label, Status), ranken und projizieren es mit dem Code, den `list` lokal ausführt; `read::list_board` liest dasselbe Brett aus einem Speicher. Die gehostete Liste wird nicht geblättert: `limit` schneidet sie mit offengelegter Gesamtzahl, `paginate` oder ein Token wird abgelehnt, wie beim gehosteten `next`. Ein gehostetes `next`, dessen Schlüssel den Rang lesen, den `[ranking.list]` einem Typ gibt, rankt jetzt ebenfalls, statt abgelehnt zu werden: `RankingReads` bekommt `list_types` und `RankingGraph` bekommt `listed` (beide `#[non_exhaustive]`, also additiv). Für den Betrieb eines Servers ändert sich der Tabellenvertrag: ein dritter dünn besetzter Index, GSI `typed` — Partitionsschlüssel `nxf_typed` (String, `#`), Sortierschlüssel `sk` (String), Projektion ALL —, den der Fold an jeder lebenden Item-Zeile setzt (geschlossene und archivierte eingeschlossen, beim Löschen entfernt). Zuerst den Index in den Stack aufnehmen, dann das Lesen ausrollen; ein Fold gegen eine Tabelle ohne den Index schreibt das Attribut folgenlos. Er kommt mit der Fold-Index-Revision 3 zusammen mit den selbstbeschreibenden Adjazenzeinträgen, ein Neufalten deckt also beides ab. diff --git a/crates/core/src/graph.rs b/crates/core/src/graph.rs index dca4ce6..34b8f84 100644 --- a/crates/core/src/graph.rs +++ b/crates/core/src/graph.rs @@ -30,7 +30,8 @@ pub struct Ticket { } /// A present edge, `from` → `to` (a `dep` points at its blocker, a `parent` edge at the parent). -#[derive(Debug, Clone, PartialEq, Eq)] +/// Ordered by `from`, `to`, `kind`, so a set of edges has one order whatever order they came in. +#[derive(Debug, Clone, PartialEq, Eq, PartialOrd, Ord)] pub struct Edge { pub from: String, pub to: String, diff --git a/crates/core/src/model.rs b/crates/core/src/model.rs index e00fb57..5beca2d 100644 --- a/crates/core/src/model.rs +++ b/crates/core/src/model.rs @@ -48,7 +48,7 @@ impl Status { /// the OR-set replacement for the former single-parent `belongs_to` LWW register. The substrate is /// n:m-capable like every other edge; single-parent is enforced at the write seam during ②a and /// relaxed in ②c. The single current parent is projected by the `present_parent` view. -#[derive(Debug, Clone, Copy, PartialEq, Eq, Hash)] +#[derive(Debug, Clone, Copy, PartialEq, Eq, PartialOrd, Ord, Hash)] pub enum EdgeKind { Dep, ContributesTo, diff --git a/crates/facade/src/read.rs b/crates/facade/src/read.rs index 47d775b..6f3ec09 100644 --- a/crates/facade/src/read.rs +++ b/crates/facade/src/read.rs @@ -24,7 +24,7 @@ use crate::plugin::{Dir, Nulls, PluginConfig, RankKey, RankSpec, Verb}; use crate::record::{item_value, items_value, raw_field}; use nexus_flow_core::derive; use nexus_flow_core::graph::Board; -use nexus_flow_core::model::{ItemRow, LinkWeight, ThreadLink}; +use nexus_flow_core::model::{EdgeKind, ItemRow, LinkWeight, ThreadLink}; use nexus_flow_core::store::Store; use serde_json::{json, Value}; use std::collections::{BTreeMap, BTreeSet, HashMap, HashSet}; @@ -2102,6 +2102,20 @@ fn blocked_record(cfg: Option<&PluginConfig>, row: &BlockedItem, j: &RecordJoins rec } +/// One `list` record without a store (6j6v.xkr7): what [`list_to_value_with_custom`] gives — the +/// canonical item, the sparse `labels` and `parent_closed_reason` joins, and the lane joins. +fn list_record(cfg: Option<&PluginConfig>, item: &ItemRow, j: &RecordJoins) -> Value { + let mut rec = item_value(item); + if !j.labels.is_empty() { + rec["labels"] = json!(j.labels); + } + if let Some(reasons) = parent_closed_reason_value(j.parent_closed) { + rec["parent_closed_reason"] = reasons; + } + put_lane_joins(cfg, &mut rec, j); + rec +} + /// One record of a bare lane (`deferred`/`closed`/`archived`/`search`): the canonical item and the /// lane joins. fn lane_record(cfg: Option<&PluginConfig>, item: &ItemRow, j: &RecordJoins) -> Value { @@ -2734,6 +2748,11 @@ pub struct ActiveBoard { pub board: Board, /// By id. pub tickets: BTreeMap, + /// The graph the derived ranking keys of `[ranking.next]` read (6j6v.1c5s): `None` until a + /// reader fills it — [`with_ranking`] from a store, `nxs_fold_ddb::records::with_ranking` from + /// the folded tables — for what [`next_ranking_reads`] names. A plugin without derived `next` + /// keys never needs it; [`next_active`] refuses one with such keys while it is missing. + pub ranking: Option, } impl ActiveBoard { @@ -2745,6 +2764,7 @@ impl ActiveBoard { .into_iter() .map(|t| (t.item.id.clone(), t)) .collect(), + ranking: None, } } @@ -2780,6 +2800,209 @@ impl ActiveBoard { } } +/// What the derived ranking keys of a `[ranking.next]` read beyond the active tickets' records +/// (6j6v.1c5s) — what a reader of an [`ActiveBoard`] puts in its [`RankingGraph`]. +/// [`next_ranking_reads`] gives it for a plugin. +/// +/// `#[non_exhaustive]`: start from `RankingReads::default()` and set the fields. +#[derive(Debug, Clone, Default, PartialEq, Eq)] +#[non_exhaustive] +pub struct RankingReads { + /// The edge kinds the keys follow or count; `mentions` too when a key counts `mentioned_by`. + pub kinds: BTreeSet, + /// A count filters on a label: the neighbours' labels. + pub labels: bool, + /// An `of` reads a custom field: the neighbours' custom values. + pub custom: bool, + /// The types whose `[ranking.list]` rank a key reads (6j6v.xkr7): the graph also holds every + /// live, unarchived item of each — the type's `list` scope ([`RankingGraph::listed`]) — and the + /// edges around them. A server reads them from its `typed` index. + pub list_types: BTreeSet, +} + +impl RankingReads { + /// Whether a graph read for `self` holds everything a graph read for `needed` holds. + pub fn covers(&self, needed: &RankingReads) -> bool { + needed.kinds.is_subset(&self.kinds) + && (self.labels || !needed.labels) + && (self.custom || !needed.custom) + && needed.list_types.is_subset(&self.list_types) + } +} + +/// The graph around a board's rows that the derived ranking keys read (6j6v.1c5s, +/// docs/specs/derived-ranking-keys.md §6): the input [`next_active`] ranks a `[ranking.next]` with +/// `related` / `count` / `sum` keys from, and [`list_board_page`] a `[ranking.list]` (6j6v.xkr7), +/// so a server ranks exactly as [`next`] and [`list_page`] do over a store. +/// +/// The SUBJECTS are the rows a key is evaluated on: the board's own rows — an [`ActiveBoard`]'s +/// active tickets; a [`ListBoard`]'s listed items and, when it carries one, its active board's +/// tickets — and the `listed` items below. +/// +/// - `edges`: every present edge of a kind in `reads.kinds` with at least one end a subject and +/// both ends LIVE (not deleted, not missing). A `parent` edge is the `present_parent` projection — +/// a child's one current parent, the one `belongs_to` reads — so a child re-parented concurrently +/// is under one parent only. Distinct, ordered. +/// - `listed`: the ids of every live, unarchived item of a type in `reads.list_types` — the `list` +/// scope a `[ranking.list]` rank is computed over (6j6v.xkr7). Empty without list types. +/// - `neighbours`: every live row the board does not hold itself — the other ends of `edges` and +/// the `listed` items, closed and archived items included, since a key reads every LIVE neighbour +/// (owner, 2026-10-11) — each with its row (`belongs_to` filled as the store fills it), its +/// labels when `reads.labels` and its non-empty custom values when `reads.custom`, else empty. +/// +/// `#[non_exhaustive]`: build it with [`RankingGraph::new`] and set the fields. +#[derive(Debug, Clone, Default, PartialEq, Eq)] +#[non_exhaustive] +pub struct RankingGraph { + /// What the graph was read for. + pub reads: RankingReads, + pub edges: BTreeSet, + /// By id. + pub neighbours: BTreeMap, + /// The live, unarchived items of `reads.list_types` (6j6v.xkr7), by id. + pub listed: BTreeSet, +} + +impl RankingGraph { + /// An empty graph read for `reads`. + pub fn new(reads: RankingReads) -> RankingGraph { + RankingGraph { + reads, + ..RankingGraph::default() + } + } +} + +/// A live item a derived ranking key reads that the board does not hold itself — next to one of +/// its rows, or in the `list` scope of a type ([`RankingGraph::neighbours`]). +/// +/// `#[non_exhaustive]`: build it with [`Neighbour::new`] and set the fields. +#[derive(Debug, Clone, PartialEq, Eq)] +#[non_exhaustive] +pub struct Neighbour { + pub item: ItemRow, + /// Present labels, sorted and distinct, as `present_labels`. + pub labels: Vec, + /// Every non-empty custom value, as the store reads them in bulk. + pub custom: BTreeMap, +} + +impl Neighbour { + /// A neighbour with no labels or custom values read. + pub fn new(item: ItemRow) -> Neighbour { + Neighbour { + item, + labels: Vec::new(), + custom: BTreeMap::new(), + } + } +} + +/// What the store-free `next` of `cfg` reads beyond the active tickets' records — the +/// [`RankingReads`] a reader fills [`ActiveBoard::ranking`] for — or `None` when `[ranking.next]` +/// declares no derived ranking key (6j6v.1c5s). It does not depend on the board. +/// +/// A `[ranking.next]` that reads, directly or down its type chain, the rank of a type only +/// `[ranking.list]` orders names that type in [`RankingReads::list_types`] (6j6v.xkr7): that rank +/// is computed over the type's live items, which the reader puts in the graph. Until 6j6v.xkr7 +/// such a plugin was refused here; it is no longer, and the `Result` stays for a later refusal. +pub fn next_ranking_reads(cfg: &PluginConfig) -> Result> { + derived::reads_for_next(cfg) +} + +/// `board` with its [`RankingGraph`] for `reads` read from `store` (6j6v.1c5s) — the parity oracle +/// for `nxs_fold_ddb::records::with_ranking`, as [`active_board`] is for the records. One edge pass +/// ([`Store::present_edges_of_kinds`]), one bulk read of the neighbours' rows, and one of their +/// labels and of their custom values when `reads` asks for them; with list types, one read of the +/// live items (6j6v.xkr7). +pub fn with_ranking( + store: &Store, + mut board: ActiveBoard, + reads: &RankingReads, +) -> Result { + let held: BTreeSet<&str> = board.tickets.keys().map(String::as_str).collect(); + board.ranking = Some(ranking_graph(store, reads, &held)?); + Ok(board) +} + +/// The [`RankingGraph`] for `reads` around the rows a board holds itself (`held`, every one a +/// subject), read from `store` — what [`with_ranking`] and [`list_board`] put on their boards. +fn ranking_graph( + store: &Store, + reads: &RankingReads, + held: &BTreeSet<&str>, +) -> Result { + let listed: Vec = if reads.list_types.is_empty() { + Vec::new() + } else { + derived::list_scope(store)? + .into_iter() + .filter(|i| { + i.item_type + .as_deref() + .is_some_and(|t| reads.list_types.contains(t)) + }) + .collect() + }; + let listed_ids: BTreeSet<&str> = listed.iter().map(|i| i.id.as_str()).collect(); + let subject = |id: &str| held.contains(id) || listed_ids.contains(id); + let kinds: Vec = reads.kinds.iter().copied().collect(); + let touching: Vec<(String, String, EdgeKind)> = store + .present_edges_of_kinds(&kinds)? + .into_iter() + .filter(|(from, to, _)| subject(from) || subject(to)) + .collect(); + let others: BTreeSet<&str> = touching + .iter() + .flat_map(|(from, to, _)| [from.as_str(), to.as_str()]) + .filter(|id| !subject(id)) + .collect(); + let others: Vec<&str> = others.into_iter().collect(); + let live: BTreeMap = store + .get_items(&others)? + .into_iter() + .filter(|i| i.deleted.as_deref() != Some("1")) + .map(|i| (i.id.clone(), i)) + .collect(); + let edges: BTreeSet = touching + .into_iter() + .filter(|(from, to, _)| { + (subject(from) || live.contains_key(from)) && (subject(to) || live.contains_key(to)) + }) + .map(|(from, to, kind)| nexus_flow_core::graph::Edge { from, to, kind }) + .collect(); + let mut rows = live; + for item in &listed { + if !held.contains(item.id.as_str()) { + rows.insert(item.id.clone(), item.clone()); + } + } + let ids: Vec<&str> = rows.keys().map(String::as_str).collect(); + let mut labels = if reads.labels { + store.labels_of_bulk(&ids)? + } else { + BTreeMap::new() + }; + let mut custom = if reads.custom { + store.custom_fields_of_bulk(&ids)? + } else { + BTreeMap::new() + }; + let mut graph = RankingGraph::new(reads.clone()); + graph.edges = edges; + graph.listed = listed_ids.iter().map(|id| id.to_string()).collect(); + graph.neighbours = rows + .into_iter() + .map(|(id, item)| { + let mut n = Neighbour::new(item); + n.labels = labels.remove(&id).unwrap_or_default(); + n.custom = custom.remove(&id).unwrap_or_default(); + (id, n) + }) + .collect(); + Ok(graph) +} + /// The store path's input as an [`ActiveBoard`]: the store's selection ([`graph::select`]) and, for /// every active ticket, its row and joins, each read in bulk. What [`next_active`] and the other /// store-free lanes answer from it is what [`next`] and its siblings answer from the store. @@ -2823,21 +3046,45 @@ pub fn active_board(store: &Store) -> Result { /// /// [`Engine::next`]: crate::engine::Engine::next /// -/// A `[ranking.next]` with derived ranking keys (6j6v.x0wf) is refused with `validation`: an -/// [`ActiveBoard`] holds neither the edges nor the scopes those keys read, and ranking without them -/// would order apart from the store path in silence. The server path is its own cut (6j6v.1c5s). +/// A `[ranking.next]` with derived ranking keys (6j6v.x0wf) is ranked from the board's +/// [`RankingGraph`] (6j6v.1c5s), by the same evaluation the store path runs — a key that reads the +/// rank of a type only `[ranking.list]` orders too, from the graph's listed items (6j6v.xkr7). It +/// is refused with `validation` when the board carries no graph, or one read for less than +/// [`next_ranking_reads`] names: ranking without those edges would order apart from the store path +/// in silence. pub fn next_active(cfg: &PluginConfig, board: &ActiveBoard, now: &str) -> Result> { crate::validate::iso_date(now)?; - if cfg.ranking.has_derived(Verb::Next) { - return Err(NxfError::validation(format!( - "plugin '{}' ranks `next` by derived ranking keys, which the store-free lanes do not \ - compute yet — answer `next` from a store", - cfg.name - ))); - } + let graph = match next_ranking_reads(cfg)? { + None => None, + Some(needed) => Some( + board + .ranking + .as_ref() + .filter(|g| g.reads.covers(&needed)) + .ok_or_else(|| { + NxfError::validation(format!( + "plugin '{}' ranks `next` by derived ranking keys, and the board was read \ + without the graph they read — fill `ActiveBoard::ranking` for \ + `next_ranking_reads` (`read::with_ranking` from a store, \ + `records::with_ranking` from the folded tables)", + cfg.name + )) + })?, + ), + }; let candidates = board.board.lanes(now).candidates; let items = board.items(candidates.iter().map(|c| &c.id)); - Ok(next_lane(cfg, candidates, items, DEFAULT_SORT_NEXT, None)) + let derived = match graph { + Some(graph) => derived::compute_active(cfg, board, graph, &items)?, + None => None, + }; + Ok(next_lane( + cfg, + candidates, + items, + DEFAULT_SORT_NEXT, + derived.as_ref(), + )) } /// [`next_active`], narrowed by `filter` (6j6v.15ed): the rows [`next_filtered`] gives over a store @@ -2929,6 +3176,267 @@ pub fn deferred_active_value(cfg: &PluginConfig, board: &ActiveBoard, now: &str) Ok(board.records(&items, |i, j| lane_record(Some(cfg), i, j))) } +// ---- list without a store (6j6v.xkr7) ---------------------------------------- +// +// The hosted `list`: the same rows in the same order as [`list_page`] over a store holding the same +// board, from a [`ListBoard`] a server reads off its `typed` index (docs/specs/derived-ranking-keys.md +// §6) — the listed types' live items with their joins, `next`'s board when a key reads a rank +// `next` gives, and the [`RankingGraph`] the derived keys read. As for the store-free `next`, the +// facade decides what to read ([`list_reads`]) and ranks and projects it; the server only reads. + +/// What the store-free `list` a [`ListQuery`] asks for reads (6j6v.xkr7) — the input of a reader +/// that builds the [`ListBoard`] ([`list_board`] from a store, `nxs_fold_ddb::records::list_board` +/// from the folded tables). [`list_reads`] gives it. +/// +/// `#[non_exhaustive]`: start from `ListReads::default()` and set the fields. +#[derive(Debug, Clone, Default, PartialEq, Eq)] +#[non_exhaustive] +pub struct ListReads { + /// The listed types: the board holds every live, unarchived item of each. `None`: every live, + /// unarchived item, whatever its type — a query without a type filter. + pub types: Option>, + /// A key reads the rank of a type `next` ranks: the board carries `next`'s board, the active + /// tickets and their [`Board`], to take `next`'s candidates from at the query's `now`. + pub next: bool, + /// The graph the derived keys read (`None`: none in play). Its subjects are the listed items, + /// the active tickets when `next` is set, and its own listed items + /// ([`RankingReads::list_types`]). + pub ranking: Option, +} + +/// One listed item and the joins its `list` record and the filters read (6j6v.xkr7) — the +/// per-item input of the store-free `list`. Every field holds what the store path reads for the +/// same item: `item` with its `belongs_to` (`present_parent`), `labels` as `present_labels`, +/// `custom` every non-empty custom value, the timestamps with an unstamped `''` as `None`, +/// `contributes_to` the present targets sorted and distinct, and `parent_closed` what +/// [`parent_closed_reason`] gives. +/// +/// `#[non_exhaustive]`: build it with [`ListItem::new`] and set the fields. +#[derive(Debug, Clone, PartialEq, Eq)] +#[non_exhaustive] +pub struct ListItem { + pub item: ItemRow, + pub labels: Vec, + pub custom: BTreeMap, + pub created_at: Option, + pub updated_at: Option, + pub contributes_to: Vec, + pub parent_closed: Vec<(String, Option)>, +} + +impl ListItem { + /// An item with no joins yet. + pub fn new(item: ItemRow) -> ListItem { + ListItem { + item, + labels: Vec::new(), + custom: BTreeMap::new(), + created_at: None, + updated_at: None, + contributes_to: Vec::new(), + parent_closed: Vec::new(), + } + } + + fn joins(&self) -> RecordJoins<'_> { + RecordJoins { + labels: &self.labels, + parent_closed: &self.parent_closed, + created_at: self.created_at.as_deref(), + updated_at: self.updated_at.as_deref(), + custom: Some(&self.custom), + ..RecordJoins::default() + } + } +} + +/// The input of the store-free `list` (6j6v.xkr7): what a reader read for a [`ListReads`]. +/// +/// `#[non_exhaustive]`: build it with [`ListBoard::new`] and set the fields. +#[derive(Debug, Clone, Default)] +#[non_exhaustive] +pub struct ListBoard { + /// The types `items` holds every live, unarchived item of; `None`: every type. + pub types: Option>, + /// The listed items, by id: live and unarchived, of `types`. + pub items: BTreeMap, + /// `next`'s board, for a key that reads the rank of a type `next` ranks + /// ([`ListReads::next`]): the active tickets' rows — their labels and custom values when the + /// graph reads them — and their [`Board`]. Its own `ranking` stays `None`. + pub active: Option, + /// The graph the derived keys read ([`ListReads::ranking`]). + pub ranking: Option, +} + +impl ListBoard { + /// The listed items of `types` (`None`: every type), keyed by their ids. + pub fn new( + types: Option>, + items: impl IntoIterator, + ) -> ListBoard { + ListBoard { + types, + items: items.into_iter().map(|i| (i.item.id.clone(), i)).collect(), + active: None, + ranking: None, + } + } +} + +/// What the store-free `list` `q` reads under `cfg` (6j6v.xkr7): the listed types — `q`'s type +/// filter, else every type —, `next`'s board when a `[ranking.list]` key reads a rank `next` gives +/// and `q` has a `now`, and the graph the derived keys of a `sort = rank` read. It depends on the +/// query, not on the board. +pub fn list_reads(cfg: &PluginConfig, q: &ListQuery) -> ListReads { + derived::reads_for_list(cfg, q) +} + +/// The store's [`ListBoard`] for `reads` — the parity oracle for +/// `nxs_fold_ddb::records::list_board`, as [`active_board`] is for the records: the listed items +/// with their joins read in bulk, `next`'s board from the store's selection, and the graph. +pub fn list_board(store: &Store, reads: &ListReads) -> Result { + let items: Vec = derived::list_scope(store)? + .into_iter() + .filter(|i| { + reads + .types + .as_ref() + .is_none_or(|types| i.item_type.as_deref().is_some_and(|t| types.contains(t))) + }) + .collect(); + let ids: Vec<&str> = items.iter().map(|i| i.id.as_str()).collect(); + let mut labels = store.labels_of_bulk(&ids)?; + let mut custom = store.custom_fields_of_bulk(&ids)?; + let mut ts = store.item_timestamps_of_bulk(&ids)?; + let mut contributes = store.contributes_to_of_bulk(&ids)?; + let mut listed = Vec::with_capacity(items.len()); + for item in items { + let id = item.id.clone(); + let parent_closed = parent_closed_reason(store, &item)?; + let (created_at, updated_at) = ts.remove(&id).unwrap_or_default(); + let mut i = ListItem::new(item); + i.labels = labels.remove(&id).unwrap_or_default(); + i.custom = custom.remove(&id).unwrap_or_default(); + i.created_at = created_at; + i.updated_at = updated_at; + i.contributes_to = contributes.remove(&id).unwrap_or_default(); + i.parent_closed = parent_closed; + listed.push(i); + } + let mut board = ListBoard::new(reads.types.clone(), listed); + let graph_reads = reads.ranking.clone().unwrap_or_default(); + if reads.next { + let selection = nexus_flow_core::graph::select(store.connection())?; + let ids: Vec<&str> = selection.tickets().map(|t| t.id.as_str()).collect(); + let items = store.get_items(&ids)?; + let mut labels = if graph_reads.labels { + store.labels_of_bulk(&ids)? + } else { + BTreeMap::new() + }; + let mut custom = if graph_reads.custom { + store.custom_fields_of_bulk(&ids)? + } else { + BTreeMap::new() + }; + let tickets = items.into_iter().map(|item| { + let id = item.id.clone(); + let mut t = ActiveTicket::new(item); + t.labels = labels.remove(&id).unwrap_or_default(); + t.custom = custom.remove(&id).unwrap_or_default(); + t + }); + board.active = Some(ActiveBoard::new(selection, tickets)); + } + if let Some(r) = &reads.ranking { + let mut held: BTreeSet<&str> = board.items.keys().map(String::as_str).collect(); + if let Some(active) = &board.active { + held.extend(active.tickets.keys().map(String::as_str)); + } + let graph = ranking_graph(store, r, &held)?; + board.ranking = Some(graph); + } + Ok(board) +} + +/// [`list_page`] without a store (6j6v.xkr7): the rows of `board` that `q` keeps — its `status`, +/// its filter through the same membership rule ([`NextFilter`]) — ordered by `q.sort`, with the +/// derived keys of a `sort = rank` from the board's graph, and cut to `q.limit` with the total +/// disclosed. The same rows in the same order as [`list_page`] over a store holding the same board. +/// +/// Not paged: the store path's page cache is machine-local, and a server's paging is an open +/// decision (6j6v.15ed) — as for [`next_active_filtered`], a server cuts this list itself. So +/// `validation` for `q.paginate` or a token; and for a malformed `q.now`, a type filter (or none) +/// the board does not list every item for, and a board read for less than the ranking needs. +pub fn list_board_page(cfg: &PluginConfig, board: &ListBoard, q: &ListQuery) -> Result { + if q.paginate || q.token.is_some() { + return Err(NxfError::validation( + "the store-free `list` is not paged — a server cuts it itself; drop paginate and the \ + token", + )); + } + if let Some(now) = &q.now { + crate::validate::iso_date(now)?; + } + let listed = match &board.types { + None => true, + Some(types) => { + !q.filter.types.is_empty() && q.filter.types.iter().all(|t| types.contains(t)) + } + }; + if !listed { + return Err(NxfError::validation( + "the list board was read for other types than the query lists — read it with \ + `list_reads` for this query", + )); + } + let mut items: Vec = board + .items + .values() + // The board holds live, unarchived items; a caller-built one is held to the same rule. + .filter(|i| i.item.deleted.as_deref() != Some("1") && i.item.archived.is_none()) + .filter(|i| { + q.status + .as_deref() + .is_none_or(|s| i.item.status.as_deref() == Some(s)) + }) + .filter(|i| q.filter.keeps(&i.item, &i.labels, &i.contributes_to)) + .map(|i| i.item.clone()) + .collect(); + let key = q.sort.unwrap_or(DEFAULT_SORT_LIST); + if key == SortKey::Rank { + let derived = derived::compute_list(cfg, board, &items, q.now.as_deref())?; + let rank = TypeRank::list(cfg).with_derived(derived.as_ref()); + items.sort_by(|a, b| rank.cmp(a, b)); + } else { + order_by(cfg, &mut items, key); + } + Ok(truncate_next(items, q.limit)) +} + +/// [`list_board_page`] as the `list` JSON envelope `{"items", "total"}` — the value +/// [`Engine::list_query_value`] gives for the same board and an unpaged query, every record with +/// its joins from the item's [`ListItem`]. +/// +/// [`Engine::list_query_value`]: crate::engine::Engine::list_query_value +pub fn list_board_value(cfg: &PluginConfig, board: &ListBoard, q: &ListQuery) -> Result { + let page = list_board_page(cfg, board, q)?; + let records = Value::Array( + page.items + .iter() + .map(|i| { + let joins = board + .items + .get(&i.id) + .map(ListItem::joins) + .unwrap_or_default(); + list_record(Some(cfg), i, &joins) + }) + .collect(), + ); + Ok(next_page_envelope(records, &page, false)) +} + // ---- lane-ranked search (C6 #916.6) ---------------------------------------- // // `search` is a facade method (not CLI-only) so the CLI and an in-process consumer share ONE diff --git a/crates/facade/src/read/derived.rs b/crates/facade/src/read/derived.rs index 86e3e3a..7c8d8ad 100644 --- a/crates/facade/src/read/derived.rs +++ b/crates/facade/src/read/derived.rs @@ -19,10 +19,27 @@ //! //! So a query costs O(E + N log N) in the relevant edges and the items in scope, and a plugin //! that declares no derived key pays nothing: [`compute`] returns `None` before any read. +//! +//! **Without a store** (6j6v.1c5s). The store-free `next` ([`super::next_active`]) ranks a board a +//! server read: [`compute_active`] takes steps 1 and 2 from the board's [`RankingGraph`] — the +//! edges, already projected and filtered to live ends, and the live neighbours' rows, labels and +//! custom values — and runs steps 3 and 4 on the same [`Context`], so the two paths cannot evaluate +//! a key apart. A neighbour counts when it is LIVE, closed and archived ones included, on both +//! paths (spec §6, owner 2026-10-11). What the graph must hold is [`reads_for_next`]. +//! +//! The store-free `list` (6j6v.xkr7, [`super::list_board_page`]) ranks a [`ListBoard`] the same +//! way: [`compute_list`] takes the listed rows from the board, `next`'s candidates — when a key +//! reads a rank `next` gives — from its active board, and every other row from its graph. A +//! `[ranking.list]` rank a key reads is computed over the type's live items, which the graph holds +//! for every type in [`RankingReads::list_types`] (the `typed` index on a server), so a `next` key +//! may read one too ([`compute_active`]). What a `list` reads is [`reads_for_list`]. -use super::{lanes, mentioners_of, resolve_items}; +use super::{ + lanes, mentioners_of, resolve_items, ActiveBoard, ListBoard, ListQuery, ListReads, + RankingGraph, RankingReads, SortKey, +}; use crate::error::{NxfError, Result}; -use crate::plugin::{Dir, Nulls, Pick, PluginConfig, RankKey, RankOf, Verb}; +use crate::plugin::{Dir, Nulls, Pick, PluginConfig, RankKey, RankOf, RankSpec, Verb}; use crate::record::{raw_field, RANKABLE_FIELDS}; use nexus_flow_core::model::{EdgeKind, ItemRow}; use nexus_flow_core::store::Store; @@ -131,20 +148,69 @@ pub(super) fn list_scope(store: &Store) -> Result> { .collect()) } -/// The values the derived keys of `verb`'s policy rank `ranked` by — `None`, before any read, -/// when the policy has no derived key. `validation` when a `list` key reads the rank of a type -/// `next` ranks and the caller passed neither `next`'s candidates nor a reference time. -pub(super) fn compute( - cfg: &PluginConfig, - store: &Store, - verb: Verb, - ranked: &[ItemRow], - scopes: Scopes<'_>, -) -> Result> { +/// A live row and its joins, wherever a store-free board holds it: an active ticket, a listed item, +/// or a neighbour in the graph. +#[derive(Clone, Copy)] +pub(super) struct Joined<'b> { + item: &'b ItemRow, + labels: &'b [String], + custom: &'b BTreeMap, +} + +/// The row `id` of an [`ActiveBoard`] with its `graph`: an active ticket's, else a neighbour's. +fn active_joined<'b>( + board: &'b ActiveBoard, + graph: &'b RankingGraph, + id: &str, +) -> Option> { + match board.tickets.get(id) { + Some(t) => Some(Joined { + item: &t.item, + labels: &t.labels, + custom: &t.custom, + }), + None => graph.neighbours.get(id).map(|n| Joined { + item: &n.item, + labels: &n.labels, + custom: &n.custom, + }), + } +} + +/// The row `id` of a [`ListBoard`] with its `graph`: a listed item's, else an active ticket's, +/// else a neighbour's. +fn list_joined<'b>(board: &'b ListBoard, graph: &'b RankingGraph, id: &str) -> Option> { + if let Some(i) = board.items.get(id) { + return Some(Joined { + item: &i.item, + labels: &i.labels, + custom: &i.custom, + }); + } + match &board.active { + Some(active) => active_joined(active, graph, id), + None => graph.neighbours.get(id).map(|n| Joined { + item: &n.item, + labels: &n.labels, + custom: &n.custom, + }), + } +} + +/// The derived side of one query, decided before any read: the type chain the ranked rows' orders +/// read the rank of (leaves first), and what the derived keys in play read. +struct Plan<'c> { + spec: &'c RankSpec, + chain: Vec, + needs: Needs, +} + +/// The [`Plan`] for ranking `ranked` under `verb` — `None` when the policy has no derived key. +fn plan<'c>(cfg: &'c PluginConfig, verb: Verb, ranked: &[ItemRow]) -> Option> { let ranking = &cfg.ranking; let spec = ranking.spec(verb); if !spec.has_derived() { - return Ok(None); + return None; } // The chain, leaves first: the orders the ranked rows' orders read the rank of, transitively. @@ -158,10 +224,128 @@ pub(super) fn compute( } } + // Every derived key in play. + let mut keys: Vec<&RankKey> = Vec::new(); + for ty in &ranked_types { + keys.extend(spec.order_for(*ty).iter().filter(|k| k.is_derived())); + } + for (v, ty) in &chain { + keys.extend( + ranking.spec(*v).by_type[ty] + .order + .iter() + .filter(|k| k.is_derived()), + ); + } + let needs = Needs::of(&keys); + Some(Plan { spec, chain, needs }) +} + +impl Plan<'_> { + /// The rows each order of the chain ranks: its type's rows in the scope of its verb. + fn node_rows<'r>(&self, next: &'r [ItemRow], list: &'r [ItemRow]) -> Vec> { + self.chain + .iter() + .map(|(v, ty)| { + let scope = match v { + Verb::Next => next, + Verb::List => list, + }; + scope + .iter() + .filter(|i| i.item_type.as_deref() == Some(ty.as_str())) + .collect() + }) + .collect() + } + + /// Whether some order of the chain is a `[ranking.list]` order. + fn reads_list(&self) -> bool { + self.chain.iter().any(|(v, _)| *v == Verb::List) + } + + /// Whether some order of the chain is a `[ranking.next]` order. + fn reads_next(&self) -> bool { + self.chain.iter().any(|(v, _)| *v == Verb::Next) + } + + /// What a [`RankingGraph`] must hold for this plan: its keys' reads, and the live items of + /// every `[ranking.list]` type of the chain the board does not list itself (`held`). + fn reads(&self, held: impl Fn(&str) -> bool) -> RankingReads { + let mut reads = self.needs.reads(); + reads.list_types = self + .chain + .iter() + .filter(|(v, ty)| *v == Verb::List && !held(ty)) + .map(|(_, ty)| ty.clone()) + .collect(); + reads + } +} + +/// The rows of a graph's listed items (the `list` scope of [`RankingReads::list_types`]), looked +/// up through `rows` — every one of them is a live row the board holds. +fn listed_rows<'b>( + graph: &RankingGraph, + rows: impl Fn(&str) -> Option>, +) -> Vec { + graph + .listed + .iter() + .filter_map(|id| rows(id).map(|j| j.item.clone())) + .collect() +} + +/// The graph a store-free query ranks with: the board's, when it was read for `needed`; an empty +/// one when the keys in play read nothing beyond the rows. `validation` otherwise — ranking without +/// those edges would order apart from the store path in silence. +fn graph_for<'g>( + graph: Option<&'g RankingGraph>, + needed: &RankingReads, + empty: &'g RankingGraph, + what: &str, +) -> Result<&'g RankingGraph> { + if *needed == RankingReads::default() { + return Ok(graph.unwrap_or(empty)); + } + graph.filter(|g| g.reads.covers(needed)).ok_or_else(|| { + NxfError::validation(format!( + "{what} ranks by derived ranking keys, and the board was read without the graph they \ + read — fill its `ranking` for the reads its plugin names" + )) + }) +} + +/// Every row a key is evaluated on: the ranked rows and the rows of the chain's scopes. +fn subjects<'a>( + ranked: &'a [ItemRow], + node_rows: &[Vec<&'a ItemRow>], +) -> HashMap<&'a str, &'a ItemRow> { + let mut subjects: HashMap<&str, &ItemRow> = HashMap::new(); + for row in ranked.iter().chain(node_rows.iter().flatten().copied()) { + subjects.insert(row.id.as_str(), row); + } + subjects +} + +/// The values the derived keys of `verb`'s policy rank `ranked` by — `None`, before any read, +/// when the policy has no derived key. `validation` when a `list` key reads the rank of a type +/// `next` ranks and the caller passed neither `next`'s candidates nor a reference time. +pub(super) fn compute( + cfg: &PluginConfig, + store: &Store, + verb: Verb, + ranked: &[ItemRow], + scopes: Scopes<'_>, +) -> Result> { + let Some(plan) = plan(cfg, verb, ranked) else { + return Ok(None); + }; + // The scopes the chain ranks over, read only when the caller does not hold them. let next_owned; let next_rows: &[ItemRow] = match scopes.next { - _ if !chain.iter().any(|(v, _)| *v == Verb::Next) => &[], + _ if !plan.chain.iter().any(|(v, _)| *v == Verb::Next) => &[], Some(rows) => rows, None => { let now = scopes.now.ok_or_else(|| { @@ -178,89 +362,211 @@ pub(super) fn compute( }; let list_owned; let list_rows: &[ItemRow] = match scopes.list { - _ if !chain.iter().any(|(v, _)| *v == Verb::List) => &[], + _ if !plan.reads_list() => &[], Some(rows) => rows, None => { list_owned = list_scope(store)?; &list_owned } }; - let node_rows: Vec> = chain - .iter() - .map(|(v, ty)| { - let scope = match v { - Verb::Next => next_rows, - Verb::List => list_rows, - }; - scope - .iter() - .filter(|i| i.item_type.as_deref() == Some(ty.as_str())) - .collect() - }) + let node_rows = plan.node_rows(next_rows, list_rows); + let mut ctx = Context::new(cfg, subjects(ranked, &node_rows)); + ctx.read_edges(store, &plan.needs)?; + Ok(Some(ctx.finish(&plan, verb, ranked, &node_rows))) +} + +/// [`compute`] for the store-free `next` (6j6v.1c5s): the values the derived keys of +/// `[ranking.next]` rank `candidates` — every candidate of the board — by, read from the +/// [`RankingGraph`] a reader filled instead of from a store. The candidates are `next`'s scope for +/// "rank", as on the store path; the `list` scope of a `[ranking.list]` type the chain reaches is +/// the graph's listed items (6j6v.xkr7). `None` when the policy has no derived key. +/// +/// The same [`Context`] evaluates the keys as on the store path; only where its edges, rows, +/// labels and custom values come from differs ([`Context::read_graph`]). `validation` when the graph +/// was read for less than the plan needs — a `[ranking.list]` type of the chain missing from +/// [`RankingReads::list_types`] among it. +pub(super) fn compute_active( + cfg: &PluginConfig, + board: &ActiveBoard, + graph: &RankingGraph, + candidates: &[ItemRow], +) -> Result> { + let Some(plan) = plan(cfg, Verb::Next, candidates) else { + return Ok(None); + }; + let needed = plan.reads(|_| false); + let graph = graph_for( + Some(graph), + &needed, + graph, + &format!("plugin '{}' (`next`)", cfg.name), + )?; + let rows = |id: &str| active_joined(board, graph, id); + let listed = listed_rows(graph, rows); + let node_rows = plan.node_rows(candidates, &listed); + let mut ctx = Context::new(cfg, subjects(candidates, &node_rows)); + ctx.read_graph(rows, graph, &plan.needs); + Ok(Some(ctx.finish(&plan, Verb::Next, candidates, &node_rows))) +} + +/// [`compute`] for the store-free `list` (6j6v.xkr7): the values the derived keys of +/// `[ranking.list]` rank `ranked` by, from a [`ListBoard`]. The `list` scope of a type is its live, +/// unarchived items — the board's listed items for the types it lists, the graph's listed items +/// for the others; `next`'s scope, when the chain reaches a `[ranking.next]` order, is the +/// candidates of the board's active board at `now`. `None` when the policy has no derived key. +/// +/// `validation` as on the store path when the chain reads a `next` rank and `now` is missing, and +/// when the board was read for less than the plan needs: no active board for such a chain, a +/// graph read for fewer reads, a chain type neither listed nor in the graph. +pub(super) fn compute_list( + cfg: &PluginConfig, + board: &ListBoard, + ranked: &[ItemRow], + now: Option<&str>, +) -> Result> { + let Some(plan) = plan(cfg, Verb::List, ranked) else { + return Ok(None); + }; + let held = |ty: &str| board.types.as_ref().is_none_or(|t| t.contains(ty)); + let needed = plan.reads(held); + let empty = RankingGraph::default(); + let graph = graph_for( + board.ranking.as_ref(), + &needed, + &empty, + &format!("plugin '{}' (`list`)", cfg.name), + )?; + let next_owned; + let next_rows: &[ItemRow] = if plan.reads_next() { + let now = now.ok_or_else(|| { + NxfError::validation( + "the plugin's [ranking.list] reads the rank of a type that `next` ranks, which is \ + computed over `next`'s candidates at a reference time — pass `now`", + ) + })?; + let active = board.active.as_ref().ok_or_else(|| { + NxfError::validation(format!( + "plugin '{}' ranks `list` by the rank of a type that `next` ranks, and the board \ + was read without the active board that rank is computed over — read it with \ + `ListReads::next`", + cfg.name + )) + })?; + let candidates = active.board.lanes(now).candidates; + next_owned = active.items(candidates.iter().map(|c| &c.id)); + &next_owned + } else { + &[] + }; + let rows = |id: &str| list_joined(board, graph, id); + let list_owned: Vec = if plan.reads_list() { + let mut scope: Vec = board.items.values().map(|i| i.item.clone()).collect(); + scope.extend( + listed_rows(graph, rows) + .into_iter() + .filter(|i| !board.items.contains_key(&i.id)), + ); + scope + } else { + Vec::new() + }; + let node_rows = plan.node_rows(next_rows, &list_owned); + let mut ctx = Context::new(cfg, subjects(ranked, &node_rows)); + ctx.read_graph(rows, graph, &plan.needs); + Ok(Some(ctx.finish(&plan, Verb::List, ranked, &node_rows))) +} + +/// What the store-free `next` of `cfg` reads beyond the active tickets' records (6j6v.1c5s): +/// `None` when `[ranking.next]` declares no derived key. Gathered over every order of the section +/// and of its type chain, so it does not depend on which types a board holds. Since 6j6v.xkr7 an +/// order that reads the rank of a type only `[ranking.list]` orders is no longer refused: the +/// type lands in [`RankingReads::list_types`], whose live items a reader puts in the graph. +pub(super) fn reads_for_next(cfg: &PluginConfig) -> Result> { + let spec = cfg.ranking.spec(Verb::Next); + if !spec.has_derived() { + return Ok(None); + } + let orders: Vec<&[RankKey]> = std::iter::once(spec.order.as_slice()) + .chain(spec.by_type.values().map(|o| o.order.as_slice())) .collect(); + // `next`'s own per-type orders start the chain: an order there may read another's rank. + let starts = spec.by_type.keys().map(|ty| (Verb::Next, ty.clone())); + Ok(Some(chain_reads(cfg, &orders, starts, |_| false).0)) +} - // Every row a key is evaluated on, and every derived key in play. - let mut subjects: HashMap<&str, &ItemRow> = HashMap::new(); - for row in ranked.iter().chain(node_rows.iter().flatten().copied()) { - subjects.insert(row.id.as_str(), row); +/// What the store-free `list` `q` asks for reads (6j6v.xkr7): its listed types (`None`: every +/// type), whether `next`'s board is needed, and the graph. Only `sort = rank` ranks by the +/// derived keys; any other order reads the listed items alone. +pub(super) fn reads_for_list(cfg: &PluginConfig, q: &ListQuery) -> ListReads { + let types: Option> = + (!q.filter.types.is_empty()).then(|| q.filter.types.iter().cloned().collect()); + let mut reads = ListReads { + types: types.clone(), + next: false, + ranking: None, + }; + let spec = cfg.ranking.spec(Verb::List); + if q.sort != Some(SortKey::Rank) || !spec.has_derived() { + return reads; } - let mut keys: Vec<&RankKey> = Vec::new(); - for ty in &ranked_types { - keys.extend(spec.order_for(*ty).iter().filter(|k| k.is_derived())); + let orders: Vec<&[RankKey]> = match &types { + Some(types) => types + .iter() + .map(|ty| spec.order_for(Some(ty.as_str()))) + .collect(), + None => std::iter::once(spec.order.as_slice()) + .chain(spec.by_type.values().map(|o| o.order.as_slice())) + .collect(), + }; + let starts: Vec = orders + .iter() + .flat_map(|keys| cfg.ranking.rank_deps(keys, Verb::List)) + .collect(); + let held = |ty: &str| types.as_ref().is_none_or(|t| t.contains(ty)); + let (graph, reads_next) = chain_reads(cfg, &orders, starts, held); + // Without `now` a chain that reads a `next` rank answers `validation` where a ranked row + // needs it, as on the store path — reading `next`'s board for it would be wasted. + reads.next = reads_next && q.now.is_some(); + if graph != RankingReads::default() { + reads.ranking = Some(graph); } + reads +} + +/// The reads of `orders` and of every order down the type chain from `starts`, the chain's +/// `[ranking.list]` types not `held` among them, and whether that chain holds a `[ranking.next]` +/// order. +fn chain_reads( + cfg: &PluginConfig, + orders: &[&[RankKey]], + starts: impl IntoIterator, + held: impl Fn(&str) -> bool, +) -> (RankingReads, bool) { + let mut chain: Vec = Vec::new(); + let mut seen: BTreeSet = BTreeSet::new(); + for node in starts { + post_order(cfg, node, &mut seen, &mut chain); + } + let mut keys: Vec<&RankKey> = orders + .iter() + .flat_map(|o| o.iter()) + .filter(|k| k.is_derived()) + .collect(); for (v, ty) in &chain { keys.extend( - ranking.spec(*v).by_type[ty] + cfg.ranking.spec(*v).by_type[ty] .order .iter() .filter(|k| k.is_derived()), ); } - - let needs = Needs::of(&keys); - let mut ctx = Context { - cfg, - subjects, - out: Adjacency::new(), - inb: Adjacency::new(), - mentioners: BTreeMap::new(), - neighbours: HashMap::new(), - labels: BTreeMap::new(), - custom: BTreeMap::new(), - ranks: HashMap::new(), - }; - ctx.read_edges(store, &needs)?; - - // Bottom-up: each order of the chain ranks its scope once; its dependents look the rank up. - for (node, rows) in chain.iter().zip(&node_rows) { - let keys = &ranking.spec(node.0).by_type[&node.1].order; - let values: Vec> = rows.iter().map(|r| ctx.values(keys, node.0, r)).collect(); - let mut order: Vec = (0..rows.len()).collect(); - let cmp = |a: usize, b: usize| { - super::keys_cmp_values(rows[a], rows[b], keys, Some(&values[a]), Some(&values[b])) - }; - order.sort_by(|a, b| cmp(*a, *b).then_with(|| rows[*a].id.cmp(&rows[*b].id))); - // A DENSE rank over the declared keys (spec §4): rows equal on every key share a rank, and - // the next distinct row takes the next integer. The id orders the sort, never the rank. - let mut ranks: HashMap = HashMap::with_capacity(rows.len()); - let mut rank = 0_i64; - for (n, &i) in order.iter().enumerate() { - if n > 0 && cmp(order[n - 1], i) != Ordering::Equal { - rank += 1; - } - ranks.insert(rows[i].id.clone(), rank); - } - ctx.ranks.insert(node.clone(), ranks); - } - - let values = ranked + let mut reads = Needs::of(&keys).reads(); + reads.list_types = chain .iter() - .map(|r| { - let keys = spec.order_for(r.item_type.as_deref()); - (r.id.clone(), ctx.values(keys, verb, r)) - }) + .filter(|(v, ty)| *v == Verb::List && !held(ty)) + .map(|(_, ty)| ty.clone()) .collect(); - Ok(Some(Derived { values })) + (reads, chain.iter().any(|(v, _)| *v == Verb::Next)) } /// Depth-first post-order over the type chain: every order a node reads the rank of lands before @@ -319,6 +625,21 @@ impl Needs { } } +impl Needs { + /// These needs as the [`RankingReads`] of a graph: `mentions` among the kinds when a count reads + /// `mentioned_by`. No list types; the caller adds them. + fn reads(&self) -> RankingReads { + let mut reads = RankingReads::default(); + reads.kinds.extend(self.kinds.iter().copied()); + if self.mentioned_by { + reads.kinds.insert(EdgeKind::Mentions); + } + reads.labels = self.labels; + reads.custom = self.custom; + reads + } +} + /// One count, from a `count` key or a `sum` summand. struct Count<'a> { kind: &'a str, @@ -381,7 +702,139 @@ struct Context<'a> { ranks: HashMap>, } -impl Context<'_> { +impl<'a> Context<'a> { + /// A context over `subjects`, nothing read yet. + fn new(cfg: &'a PluginConfig, subjects: HashMap<&'a str, &'a ItemRow>) -> Context<'a> { + Context { + cfg, + subjects, + out: Adjacency::new(), + inb: Adjacency::new(), + mentioners: BTreeMap::new(), + neighbours: HashMap::new(), + labels: BTreeMap::new(), + custom: BTreeMap::new(), + ranks: HashMap::new(), + } + } + + /// Rank the chain bottom-up — each order its scope once, its dependents look the rank up — and + /// then the ranked rows' values. + fn finish( + mut self, + plan: &Plan<'_>, + verb: Verb, + ranked: &[ItemRow], + node_rows: &[Vec<&ItemRow>], + ) -> Derived { + let ranking = &self.cfg.ranking; + for (node, rows) in plan.chain.iter().zip(node_rows) { + let keys = &ranking.spec(node.0).by_type[&node.1].order; + let values: Vec> = + rows.iter().map(|r| self.values(keys, node.0, r)).collect(); + let mut order: Vec = (0..rows.len()).collect(); + let cmp = |a: usize, b: usize| { + super::keys_cmp_values(rows[a], rows[b], keys, Some(&values[a]), Some(&values[b])) + }; + order.sort_by(|a, b| cmp(*a, *b).then_with(|| rows[*a].id.cmp(&rows[*b].id))); + // A DENSE rank over the declared keys (spec §4): rows equal on every key share a rank, + // and the next distinct row takes the next integer. The id orders the sort, never the + // rank. + let mut ranks: HashMap = HashMap::with_capacity(rows.len()); + let mut rank = 0_i64; + for (n, &i) in order.iter().enumerate() { + if n > 0 && cmp(order[n - 1], i) != Ordering::Equal { + rank += 1; + } + ranks.insert(rows[i].id.clone(), rank); + } + self.ranks.insert(node.clone(), ranks); + } + let values = ranked + .iter() + .map(|r| { + let keys = plan.spec.order_for(r.item_type.as_deref()); + (r.id.clone(), self.values(keys, verb, r)) + }) + .collect(); + Derived { values } + } + + /// The reads from a [`RankingGraph`] instead of a store (6j6v.1c5s): the same adjacency, reverse + /// mentions, neighbour rows, labels and custom values [`read_edges`](Self::read_edges) reads, + /// taken from the graph's edges and, through `rows`, from the board's rows — its active + /// tickets, its listed items — and the graph's neighbours. The graph holds every edge of the + /// kinds read that touches a subject with both ends live, `parent` as the `present_parent` + /// projection — the edges the store's pass yields for the same subjects. + fn read_graph<'b>( + &mut self, + rows: impl Fn(&str) -> Option>, + graph: &RankingGraph, + needs: &Needs, + ) { + let mut mention_sources: BTreeMap> = BTreeMap::new(); + for e in &graph.edges { + if needs.kinds.contains(&e.kind) { + if self.subjects.contains_key(e.to.as_str()) { + self.inb + .entry((e.kind, e.to.clone())) + .or_default() + .push(e.from.clone()); + } + if self.subjects.contains_key(e.from.as_str()) { + self.out + .entry((e.kind, e.from.clone())) + .or_default() + .push(e.to.clone()); + } + } + if needs.mentioned_by + && e.kind == EdgeKind::Mentions + && self.subjects.contains_key(e.to.as_str()) + { + mention_sources + .entry(e.to.clone()) + .or_default() + .insert(e.from.as_str()); + } + } + for (target, froms) in mention_sources { + let found: Vec = froms + .into_iter() + .filter_map(|f| rows(f).map(|j| j.item.clone())) + .collect(); + if !found.is_empty() { + self.mentioners.insert(target, found); + } + } + let missing: BTreeSet<&str> = self + .out + .values() + .chain(self.inb.values()) + .flatten() + .map(String::as_str) + .filter(|id| !self.subjects.contains_key(id)) + .collect(); + self.neighbours = missing + .into_iter() + .filter_map(|id| rows(id).map(|j| (id.to_string(), j.item.clone()))) + .collect(); + if needs.labels || needs.custom { + let mut ids: BTreeSet<&str> = self.neighbours.keys().map(String::as_str).collect(); + ids.extend(self.subjects.keys().copied()); + ids.extend(self.mentioners.values().flatten().map(|i| i.id.as_str())); + for id in ids { + let Some(j) = rows(id) else { continue }; + if needs.labels && !j.labels.is_empty() { + self.labels.insert(id.to_string(), j.labels.to_vec()); + } + if needs.custom && !j.custom.is_empty() { + self.custom.insert(id.to_string(), j.custom.clone()); + } + } + } + } + /// The reads: the edge pass, the reverse mentions, the neighbours' rows, labels and custom /// fields — each ONE bulk read (per 500 ids where the store chunks), whatever the row count. fn read_edges(&mut self, store: &Store, needs: &Needs) -> Result<()> { diff --git a/crates/facade/tests/derived_ranking.rs b/crates/facade/tests/derived_ranking.rs index ae831c4..c49f900 100644 --- a/crates/facade/tests/derived_ranking.rs +++ b/crates/facade/tests/derived_ranking.rs @@ -11,6 +11,8 @@ //! - Determinism (§9): another edge insertion order ranks alike, and ties fall to the id. //! - Bounded cost (§9, a query-shape guard in the style of xn8s): the edge pass runs once per query //! however many candidates there are. +//! - Without a store (6j6v.1c5s, 6j6v.xkr7): `read::next_active` and `read::list_board_page` over +//! the boards a server reads rank as the store path does. //! //! Its own test binary, so the fixture plugin does not pollute the facade lib tests' "exactly two //! OSS plugins" assertion. @@ -359,8 +361,37 @@ fn the_rank_follows_the_board_on_the_next_query() { } #[test] -fn the_store_free_lanes_refuse_a_plugin_with_derived_next_keys() { - // The server path is its own cut (1c5s); ranking without the edges would differ in silence. +fn the_store_free_next_ranks_from_the_ranking_graph_as_the_store_does() { + // 6j6v.1c5s: the board a server reads carries the graph the keys read, and the store-free + // `next` ranks from it exactly as the store path does — the spec's three-type chain. + let tmp = workspace("derived-rank-fixture"); + spec_board(tmp.path()); + let cfg = plugin::load("derived-rank-fixture").unwrap(); + let s = writer(tmp.path()); + let reads = read::next_ranking_reads(&cfg) + .unwrap() + .expect("derived next keys"); + assert_eq!( + reads.kinds.iter().copied().collect::>(), + [EdgeKind::ContributesTo, EdgeKind::Parent], + "the edges the next orders follow or count" + ); + assert!(reads.custom && !reads.labels, "{reads:?}"); + let board = read::with_ranking(&s, read::active_board(&s).unwrap(), &reads).unwrap(); + let got = read::next_active(&cfg, &board, NOW).unwrap(); + assert_eq!(ids(&got), ids(&read::next(&cfg, &s, NOW, None).unwrap())); + assert_eq!(of_type(&got, "project"), PROJECTS); + assert_eq!(of_type(&got, "action"), ACTIONS); + assert_eq!( + read::next_active_value(&cfg, &board, NOW).unwrap(), + read::next_to_value_with_custom(&cfg, &s, &read::next(&cfg, &s, NOW, None).unwrap()) + .unwrap() + ); +} + +#[test] +fn the_store_free_next_refuses_a_board_read_without_the_graph_its_keys_read() { + // Ranking without the edges would order apart from the store path in silence. let tmp = workspace("derived-rank-fixture"); spec_board(tmp.path()); let cfg = plugin::load("derived-rank-fixture").unwrap(); @@ -369,6 +400,336 @@ fn the_store_free_lanes_refuse_a_plugin_with_derived_next_keys() { let err = read::next_active(&cfg, &board, NOW).unwrap_err(); assert_eq!(err.kind, ErrorKind::Validation); assert!(err.msg.contains("derived ranking keys"), "{}", err.msg); + // A graph read for less than the keys need is refused too: here without the custom values + // the goals' `order` lives in. + let mut reads = read::next_ranking_reads(&cfg).unwrap().unwrap(); + reads.custom = false; + let short = read::with_ranking(&s, read::active_board(&s).unwrap(), &reads).unwrap(); + let err = read::next_active(&cfg, &short, NOW).unwrap_err(); + assert_eq!(err.kind, ErrorKind::Validation); + // `blocked` and `deferred` rank by no derived key and need no graph. + read::blocked_active(&cfg, &board); + read::deferred_active(&cfg, &board, NOW).unwrap(); +} + +/// A plugin ranking `next` by its neighbours with no status filter, so closed, archived and +/// deleted neighbours each decide something (owner, 2026-10-11: a key reads every LIVE neighbour). +const LIVE_ORDERS: &str = r#" +name = "live-neighbours-fixture" +[description] +en = "x" +de = "y" +[priority] +labels = ["P0", "P1", "P2", "P3"] +[types] +list = ["goal", "project", "action"] +[vocabulary.status] +open = "open" +in_progress = "in progress" +closed = "closed" +[ranking.next] +types = ["project", "action"] +order = [ { field = "priority", dir = "asc" } ] +[ranking.next.project] +order = [ + { related = "contributes_to", type = "goal", of = { field = "order" }, pick = "min", nulls = "last" }, + { count = "children", dir = "desc" }, + { count = "mentioned_by", dir = "desc" }, +] +[ranking.next.action] +order = [ { related = "parent", type = "project", of = "rank", nulls = "last" } ] +[presentation.list] +columns = ["id"] +[fields.order] +type = "text" +on = ["goal"] +"#; + +#[test] +fn a_closed_archived_or_deleted_neighbour_reads_alike_with_and_without_a_store() { + // Goals: `gc` closed (order 1), `ga` archived (order 2), `go` open (order 3), `gd` deleted + // (order 0). Projects `pa`..`pd` contribute to one each: a closed or archived goal still gives + // its order, a deleted one gives none (null, last). Projects `q1`..`q4` have no goal: `q1` has + // three children — two closed, one archived — `q2` one open child, `q3` is mentioned by a + // closed project and `q4` by nothing. Every child and mentioner but `q2`'s is out of `next`. + let cfg: plugin::PluginConfig = toml::from_str(LIVE_ORDERS).unwrap(); + let mut s = Store::open_in_memory(1); + for (g, order) in [ + ("zz12.gc", "1"), + ("zz12.ga", "2"), + ("zz12.go", "3"), + ("zz12.gd", "0"), + ] { + item(&mut s, g, "goal", "1"); + custom(&mut s, g, "order", order); + } + close(&mut s, "zz12.gc"); + s.set_field("zz12.ga", "archived", Some("2026-06-01".into()), "t"); + for (p, g) in [ + ("zz12.pa", "zz12.gc"), + ("zz12.pb", "zz12.ga"), + ("zz12.pc", "zz12.gd"), + ("zz12.pd", "zz12.go"), + ] { + item(&mut s, p, "project", "1"); + edge(&mut s, p, g, EdgeKind::ContributesTo); + } + s.delete_item("zz12.gd", "t"); + for q in ["zz12.q1", "zz12.q2", "zz12.q3", "zz12.q4"] { + item(&mut s, q, "project", "1"); + } + for (a, parent) in [ + ("zz12.a1", "zz12.q1"), + ("zz12.a2", "zz12.q1"), + ("zz12.a3", "zz12.q1"), + ("zz12.a4", "zz12.q2"), + ] { + item(&mut s, a, "action", "1"); + s.set_parent(a, parent, "t").unwrap(); + } + close(&mut s, "zz12.a1"); + close(&mut s, "zz12.a2"); + s.set_field("zz12.a3", "archived", Some("2026-06-01".into()), "t"); + item(&mut s, "zz12.px", "project", "1"); + edge(&mut s, "zz12.px", "zz12.q3", EdgeKind::Mentions); + close(&mut s, "zz12.px"); + + let want = [ + "zz12.pa", "zz12.pb", "zz12.pd", "zz12.q1", "zz12.q2", "zz12.q3", "zz12.pc", "zz12.q4", + ]; + let store = read::next(&cfg, &s, NOW, None).unwrap(); + assert_eq!(of_type(&store, "project"), want, "the store path"); + let reads = read::next_ranking_reads(&cfg).unwrap().unwrap(); + let board = read::with_ranking(&s, read::active_board(&s).unwrap(), &reads).unwrap(); + let graph = board.ranking.as_ref().unwrap(); + for id in ["zz12.gc", "zz12.ga", "zz12.a1", "zz12.a3", "zz12.px"] { + assert!( + graph.neighbours.contains_key(id), + "{id} is a live neighbour" + ); + } + assert!( + !graph.neighbours.contains_key("zz12.gd"), + "a deleted one is not" + ); + assert!( + graph.edges.iter().all(|e| e.to != "zz12.gd"), + "nor an edge to it" + ); + assert_eq!( + ids(&read::next_active(&cfg, &board, NOW).unwrap()), + ids(&store) + ); +} + +#[test] +fn a_next_order_reading_the_rank_of_a_type_only_list_orders_ranks_from_the_listed_items() { + // Spec §12 point 1 lets a `next` key read the rank `[ranking.list]` gives a type. That rank is + // computed over the type's live, unarchived items, which a board of active tickets does not + // hold: since 6j6v.xkr7 the reads name the type (`list_types`) and the graph holds its items — + // a server reads them off its `typed` index — so the store-free `next` ranks as the store + // path does. A board read without them is refused. + let cfg: plugin::PluginConfig = toml::from_str( + r#" + name = "list-rank-fixture" + [description] + en = "x" + de = "y" + [priority] + labels = ["P0", "P1", "P2"] + [types] + list = ["action", "person"] + [vocabulary.status] + open = "open" + in_progress = "in progress" + closed = "closed" + [ranking.next] + order = [ { field = "priority" } ] + [ranking.next.action] + order = [ { related = "mentions", type = "person", of = "rank", nulls = "last" } ] + [ranking.list.person] + order = [ { count = "mentioned_by", dir = "desc" }, { field = "priority", dir = "asc" } ] + [presentation.list] + columns = ["id"] + "#, + ) + .unwrap(); + let mut s = Store::open_in_memory(1); + // Persons: u1 mentioned twice (rank 0); u2 and the closed u3 once each, u3 with the better + // priority; u4 archived (no rank: outside the list scope); u5 deleted. + for (u, prio) in [ + ("u1", "2"), + ("u2", "1"), + ("u3", "0"), + ("u4", "0"), + ("u5", "0"), + ] { + item(&mut s, &format!("zz12.{u}"), "person", prio); + } + close(&mut s, "zz12.u3"); + s.set_field("zz12.u4", "archived", Some("2026-06-01".into()), "t"); + for (a, person) in [ + ("zz12.a1", "zz12.u2"), + ("zz12.a2", "zz12.u1"), + ("zz12.a3", "zz12.u3"), + ("zz12.a4", "zz12.u4"), + ("zz12.a5", "zz12.u5"), + ] { + item(&mut s, a, "action", "1"); + edge(&mut s, a, person, EdgeKind::Mentions); + } + item(&mut s, "zz12.a6", "action", "1"); + edge(&mut s, "zz12.a6", "zz12.u1", EdgeKind::Mentions); + s.delete_item("zz12.u5", "t"); + + let reads = read::next_ranking_reads(&cfg) + .unwrap() + .expect("derived next keys"); + assert_eq!( + reads.list_types.iter().collect::>(), + ["person"], + "{reads:?}" + ); + let board = read::with_ranking(&s, read::active_board(&s).unwrap(), &reads).unwrap(); + let graph = board.ranking.as_ref().unwrap(); + assert_eq!( + graph.listed.iter().collect::>(), + ["zz12.u1", "zz12.u2", "zz12.u3"], + "the live, unarchived persons" + ); + let store = read::next(&cfg, &s, NOW, None).unwrap(); + assert_eq!( + of_type(&store, "action"), + ["zz12.a2", "zz12.a6", "zz12.a3", "zz12.a1", "zz12.a4", "zz12.a5"] + ); + assert_eq!( + ids(&read::next_active(&cfg, &board, NOW).unwrap()), + ids(&store) + ); + + // Without the person's items the graph is short of what the keys read: refused. + let mut short = reads.clone(); + short.list_types.clear(); + let board = read::with_ranking(&s, read::active_board(&s).unwrap(), &short).unwrap(); + assert_eq!( + read::next_active(&cfg, &board, NOW).unwrap_err().kind, + ErrorKind::Validation + ); +} + +#[test] +fn the_store_free_list_ranks_persons_as_list_query_does() { + // 6j6v.xkr7: the hosted `list` — `read::list_board_page` over a `ListBoard` read for + // `read::list_reads` — gives the rows, the order, the total and the JSON `Engine::list_query` + // gives: persons by a sum of mentions, then by the rank `next` gives the projects they + // contribute to (`now` takes `next`'s candidates), filtered by type, status, container and + // label as locally. + let tmp = workspace("derived-rank-fixture"); + spec_board(tmp.path()); + let cfg = plugin::load("derived-rank-fixture").unwrap(); + let s = writer(tmp.path()); + let engine = engine(&tmp); + let queries = [ + ListQuery::new() + .sort(SortKey::Rank) + .filter(NextFilter::new().with_type("person")) + .now(NOW), + ListQuery::new() + .sort(SortKey::Rank) + .filter(NextFilter::new().with_type("person")) + .now(NOW) + .limit(2), + ListQuery::new().sort(SortKey::Rank).now(NOW), + ListQuery::new() + .sort(SortKey::Rank) + .filter( + NextFilter::new() + .with_type("action") + .with_container("zz12.p5"), + ) + .now(NOW), + ListQuery::new() + .sort(SortKey::Rank) + .filter(NextFilter::new().with_label("waiting")) + .now(NOW), + ListQuery::new().status("closed"), + ListQuery::new() + .filter(NextFilter::new().with_type("project").with_type("goal")) + .sort(SortKey::Rank), + ]; + for q in &queries { + let reads = read::list_reads(&cfg, q); + let board = read::list_board(&s, &reads).unwrap(); + let page = read::list_board_page(&cfg, &board, q).unwrap(); + let want = engine.list_query(q).unwrap(); + assert_eq!(ids(&page.items), ids(&want.items), "{q:?}"); + assert_eq!(page.total, want.total, "{q:?}"); + assert_eq!( + read::list_board_value(&cfg, &board, q).unwrap(), + engine.list_query_value(q).unwrap(), + "{q:?}" + ); + } + let persons = read::list_board_page( + &cfg, + &read::list_board(&s, &read::list_reads(&cfg, &queries[0])).unwrap(), + &queries[0], + ) + .unwrap(); + assert_eq!(ids(&persons.items), PERSONS); + let reads = read::list_reads(&cfg, &queries[0]); + assert!( + reads.next, + "persons read the rank `next` gives projects: {reads:?}" + ); + assert_eq!( + reads.types, + Some(["person".to_string()].into_iter().collect()) + ); +} + +#[test] +fn the_store_free_list_refuses_what_its_board_was_not_read_for() { + let tmp = workspace("derived-rank-fixture"); + spec_board(tmp.path()); + let cfg = plugin::load("derived-rank-fixture").unwrap(); + let s = writer(tmp.path()); + let persons = ListQuery::new() + .sort(SortKey::Rank) + .filter(NextFilter::new().with_type("person")) + .now(NOW); + let board = read::list_board(&s, &read::list_reads(&cfg, &persons)).unwrap(); + let refused = |q: &ListQuery, board: &read::ListBoard| { + let err = read::list_board_page(&cfg, board, q).unwrap_err(); + assert_eq!(err.kind, ErrorKind::Validation, "{q:?}"); + err.msg + }; + // Not paged: a server cuts the list itself. + refused(&persons.clone().limit(2).paginate(), &board); + // Other types, or every type, than the board lists. + refused( + &ListQuery::new().filter(NextFilter::new().with_type("project")), + &board, + ); + refused(&ListQuery::new(), &board); + // Without `now`, `next`'s candidates cannot be taken — as on the store path. + let no_now = ListQuery::new() + .sort(SortKey::Rank) + .filter(NextFilter::new().with_type("person")); + let reads = read::list_reads(&cfg, &no_now); + assert!(!reads.next, "nothing to read `next`'s board for: {reads:?}"); + let msg = refused(&no_now, &read::list_board(&s, &reads).unwrap()); + assert!(msg.contains("pass `now`"), "{msg}"); + // A board without the graph its keys read. + let mut short = read::list_reads(&cfg, &persons); + short.ranking = None; + let msg = refused(&persons, &read::list_board(&s, &short).unwrap()); + assert!(msg.contains("derived ranking keys"), "{msg}"); + // Without `next`'s board. + let mut short = read::list_reads(&cfg, &persons); + short.next = false; + let msg = refused(&persons, &read::list_board(&s, &short).unwrap()); + assert!(msg.contains("active board"), "{msg}"); } // ---- each kind over hand-built boards (§9 "per kind") ------------------------------------- diff --git a/crates/fold-ddb/Cargo.toml b/crates/fold-ddb/Cargo.toml index 97c413c..27dc171 100644 --- a/crates/fold-ddb/Cargo.toml +++ b/crates/fold-ddb/Cargo.toml @@ -37,3 +37,6 @@ aws-sdk-dynamodb = { version = "1", optional = true } tokio = { version = "1", features = ["rt-multi-thread", "macros", "time"] } # The records differential test (6j6v.t1ym) parses a plugin fixture that declares a custom field. toml = "0.8" +# The derived-keys differential test (6j6v.1c5s) registers its plugin fixture, so `plugin::load` +# validates it as it validates a consumer's plugin. +inventory = "0.3" diff --git a/crates/fold-ddb/src/board.rs b/crates/fold-ddb/src/board.rs index 2ffc850..d43f5d0 100644 --- a/crates/fold-ddb/src/board.rs +++ b/crates/fold-ddb/src/board.rs @@ -1,5 +1,5 @@ -//! The board's two indexes, kept beside the folded rows, and the reads that use them -//! (6j6v.vvw6 points 4 and 5). +//! The board's indexes, kept beside the folded rows, and the reads that use them +//! (6j6v.vvw6 points 4 and 5; `typed`, 6j6v.xkr7). //! //! # `active` //! @@ -27,6 +27,17 @@ //! one instant keep one order. A ticket without the instant sits under `"0"`, which sorts after //! every date — last, as the local lanes put it. [`dated`] reads one lane newest first. //! +//! # `typed` +//! +//! Sparse as well (6j6v.xkr7): a live item — not deleted, archived or not — with a type sits in +//! partition `#` ([`typed_partition`]) under its own sort key, so ONE Query lists the +//! live items of a type in id order: the `list` scope of a type the derived ranking keys rank by +//! (docs/specs/derived-ranking-keys.md §6). Like the `active` flag of a ticket it is a function of +//! the row alone, set whenever an op touches the row ([`refresh_item`]) — after the row, so a run +//! that dies in between sets it when it folds the op again. Generic: every typed item is listed, +//! whatever the plugin ranks, at one attribute per item row. A row's `typed` attribute never +//! names another row, so a retype or a delete moves nothing else. +//! //! # Who owns a label or a thread link //! //! `label_adds` and `thread_link_adds` key on the add's tag alone, so nothing in them is found by @@ -49,6 +60,19 @@ //! Only `contributes_to` adds get one: folding another edge's add costs nothing more, and folding //! an edge remove one `GetItem` of its add, to learn its kind. //! +//! # What an adjacency entry says +//! +//! Revision 3 (6j6v.1c5s) makes the `.adj##` entries self-describing, as the entries +//! under a ticket are: each copies the edge add's `from_id`, `to_id`, `kind`, `lamport` and `site` +//! and says whether the add is removed ([`REMOVED`]). Every edge add has one under each end, +//! whatever its kind and whatever the ends' activity, so ONE Query on a ticket's `.adj#` prefix +//! reads every edge touching it, in and out, with its presence — what the derived ranking keys read +//! ([`crate::records::with_ranking`]): inbound `contributes_to`, `mentions` both ways, the +//! `parent` edges a child's current parent is picked from. The entries keep their place and key; +//! only their attributes grow. Folding an edge add reads its remove (a `dep` or `parent` add read it +//! already, for its flag); folding an edge remove now reads the add and the remove and rewrites +//! both entries — two `GetItem`s and two writes, one when the edge is a loop. +//! //! # One writer per stream //! //! The rows a reducer describes converge whatever the order and however many folders write them. @@ -59,7 +83,8 @@ use crate::layout::{ adjacency_key, adjacency_prefix, contributes_adjacency_key, label_adjacency_key, - link_adjacency_key, row_key, table_prefix, ACTIVE, DATED, DATED_AT, MAX_KEY_BYTES, SK, + link_adjacency_key, row_key, table_prefix, typed_partition, ACTIVE, DATED, DATED_AT, + MAX_KEY_BYTES, SK, TYPED, }; use crate::table::{text, Dated, Row, Table}; use crate::write::{Cond, Write}; @@ -220,6 +245,29 @@ pub fn is_active(row: &Row) -> bool { ) } +/// The type a stored `items` row is listed under on the `typed` index (6j6v.xkr7): its type, when +/// it has one and is live — not deleted; archived or not. A type too long for an index partition +/// ([`MAX_PARTITION_BYTES`](crate::layout::MAX_PARTITION_BYTES)) is never folded: [`typed_partitions`] refuses its op first. +pub fn typed(row: &Row) -> Option<&str> { + if text(row, "deleted") == Some("1") { + return None; + } + text(row, "type") +} + +/// The `typed` partitions the item rows these changes write would be listed under — what +/// [`Folder`](crate::fold::Folder) checks against +/// [`MAX_PARTITION_BYTES`](crate::layout::MAX_PARTITION_BYTES) before an op writes +/// anything. +pub(crate) fn typed_partitions(stream: &str, changes: &[Change]) -> Vec { + changes + .iter() + .filter(|c| c.table == "items") + .filter_map(|c| effect_text(c, "type")) + .map(|ty| typed_partition(stream, ty)) + .collect() +} + /// The `dated` lane of a stored `items` row, and the instant it sorts by. pub fn lane(row: &Row) -> Option<(Dated, String)> { if text(row, "deleted") == Some("1") { @@ -279,7 +327,7 @@ pub(crate) async fn after(table: &T, changes: &[Change]) -> Result<(), link_edge(table, tag).await?; } for tag in removed { - refresh_edge(table, tag).await?; + link_edge(table, tag).await?; } for (at, tag) in owned_tags { refresh_owned(table, &OWNED[at], tag).await?; @@ -304,7 +352,7 @@ pub(crate) async fn refresh_item(table: &T, id: &str) -> Result<(), T: for entry in table.query_prefix(&adjacency_prefix(id)).await? { if let Some(tag) = text(&entry, "tag") { if let Some(edge) = table.get(&tag_key("edge_adds", tag)).await? { - flag_edge(table, tag, edge, Some((id, active))).await?; + flag_edge(table, tag, edge, None, Some((id, active))).await?; } } } @@ -318,6 +366,14 @@ pub(crate) async fn refresh_item(table: &T, id: &str) -> Result<(), T: remove.push(ACTIVE.to_string()); } } + let typed_now = typed(&row).map(|ty| typed_partition(table.stream(), ty)); + let typed_was = text(&row, TYPED).map(str::to_string); + if typed_now != typed_was { + match typed_now { + Some(p) => set.push((TYPED.to_string(), Cell::Text(p))), + None => remove.push(TYPED.to_string()), + } + } let dated_now = lane(&row).map(|(l, at)| (l.partition(table.stream()), dated_key(&at, id))); let dated_was = match (row.get(DATED), row.get(DATED_AT)) { (Some(Cell::Text(p)), Some(Cell::Text(at))) => Some((p.clone(), at.clone())), @@ -349,46 +405,68 @@ pub(crate) async fn refresh_item(table: &T, id: &str) -> Result<(), T: Ok(()) } -/// Record that edge `tag` touches its ends, then set its flag. -pub(crate) async fn link_edge(table: &T, tag: &str) -> Result<(), T::Error> { - let Some(edge) = table.get(&tag_key("edge_adds", tag)).await? else { - return Ok(()); - }; - let ends: BTreeSet<&str> = [text(&edge, "from_id"), text(&edge, "to_id")] +/// The cells of edge add `edge` an adjacency entry copies beside its tag and [`REMOVED`] (index +/// revision 3, 6j6v.1c5s): its ends, its kind and its coordinate. +const EDGE_COPIES: [&str; 5] = ["from_id", "to_id", "kind", "lamport", "site"]; + +/// The adjacency entries of edge add `tag` — one under each end, the same entry twice for an edge +/// from a ticket to itself — from the add's row and whether it is removed. +async fn write_adjacency( + table: &T, + tag: &str, + edge: &Row, + removed: bool, +) -> Result<(), T::Error> { + let mut set = vec![("tag".to_string(), Cell::Text(tag.to_string()))]; + for column in EDGE_COPIES { + match edge.get(column) { + None | Some(Cell::Null) => {} + Some(cell) => set.push((column.to_string(), cell.clone())), + } + } + set.push((REMOVED.to_string(), Cell::Int(i64::from(removed)))); + let ends: BTreeSet<&str> = [text(edge, "from_id"), text(edge, "to_id")] .into_iter() .flatten() .collect(); for end in ends { table - .write(&Write::put( - adjacency_key(end, tag), - vec![("tag".to_string(), Cell::Text(tag.to_string()))], - )) + .write(&Write::put(adjacency_key(end, tag), set.clone())) .await?; } - flag_edge(table, tag, edge, None).await + Ok(()) } -/// Set edge `tag`'s flag from its row, its remove and its ends. -pub(crate) async fn refresh_edge(table: &T, tag: &str) -> Result<(), T::Error> { - match table.get(&tag_key("edge_adds", tag)).await? { - Some(edge) => flag_edge(table, tag, edge, None).await, - None => Ok(()), - } +/// Record that edge `tag` touches its ends — the entry under each end says what the edge is and +/// whether it is removed — then set its flag. Run when the add is folded and again when its remove +/// is; nothing while the add has not been folded, since the add writes both. +pub(crate) async fn link_edge(table: &T, tag: &str) -> Result<(), T::Error> { + let Some(edge) = table.get(&tag_key("edge_adds", tag)).await? else { + return Ok(()); + }; + let removed = table.get(&tag_key("edge_removes", tag)).await?.is_some(); + write_adjacency(table, tag, &edge, removed).await?; + flag_edge(table, tag, edge, Some(removed), None).await } -/// `known` is a ticket whose activity the caller knows ahead of its stored flag. +/// `removed` is whether the edge's remove is folded, when the caller has read it already; `known` +/// a ticket whose activity the caller knows ahead of its stored flag. async fn flag_edge( table: &T, tag: &str, edge: Row, + removed: Option, known: Option<(&str, bool)>, ) -> Result<(), T::Error> { let indexed = matches!( text(&edge, "kind").and_then(EdgeKind::parse), Some(EdgeKind::Dep | EdgeKind::Parent) ); - let mut flag = indexed && table.get(&tag_key("edge_removes", tag)).await?.is_none(); + let mut flag = indexed + && match removed { + Some(removed) => !removed, + None => table.get(&tag_key("edge_removes", tag)).await?.is_none(), + }; if flag { let mut any_active = false; for end in [text(&edge, "from_id"), text(&edge, "to_id")] diff --git a/crates/fold-ddb/src/ddb.rs b/crates/fold-ddb/src/ddb.rs index 6089713..323d715 100644 --- a/crates/fold-ddb/src/ddb.rs +++ b/crates/fold-ddb/src/ddb.rs @@ -1,14 +1,16 @@ //! [`DynamoDbTable`]: one stream's partition of the views table in DynamoDB, and [`create_table`], //! the table the contract in the crate doc describes. //! -//! Every request names the stream's partition key; the board's two reads name an index. There is +//! Every request names the stream's partition key; the board's three index reads name an index. There is //! no Scan anywhere in this file, and the `dynamodb-local` CI job records every request a fold //! makes to prove it stays that way. //! //! The adapter is async and runs on the caller's runtime — a fold run in a Lambda awaits it //! directly, unlike the relay's op store, which bridges the SDK into its synchronous traits. -use crate::layout::{ACTIVE, ACTIVE_INDEX, DATED, DATED_AT, DATED_INDEX, PK, SK}; +use crate::layout::{ + typed_partition, ACTIVE, ACTIVE_INDEX, DATED, DATED_AT, DATED_INDEX, PK, SK, TYPED, TYPED_INDEX, +}; use crate::table::{Dated, Outcome, Row, Table}; use crate::write::Write; use aws_sdk_dynamodb::error::{DisplayErrorContext, ProvideErrorMetadata}; @@ -37,8 +39,8 @@ fn err(e: E) -> DdbError { DdbError(DisplayErrorContext(e).to_string()) } -/// The views table of `name`, as the deploying stack must provision it: keys, both sparse -/// indexes projecting every attribute, on-demand billing. Production code never sends it; the +/// The views table of `name`, as the deploying stack must provision it: keys, the three sparse +/// indexes (`active`, `dated`, `typed`) projecting every attribute, on-demand billing. Production code never sends it; the /// integration tests do, and a stack is checked against it. pub fn create_table(client: &Client, name: &str) -> CreateTableFluentBuilder { let s = |n: &str| { @@ -77,10 +79,12 @@ pub fn create_table(client: &Client, name: &str) -> CreateTableFluentBuilder { .attribute_definitions(s(ACTIVE)) .attribute_definitions(s(DATED)) .attribute_definitions(s(DATED_AT)) + .attribute_definitions(s(TYPED)) .key_schema(key(PK, KeyType::Hash)) .key_schema(key(SK, KeyType::Range)) .global_secondary_indexes(index(ACTIVE_INDEX, ACTIVE, SK)) .global_secondary_indexes(index(DATED_INDEX, DATED, DATED_AT)) + .global_secondary_indexes(index(TYPED_INDEX, TYPED, SK)) } /// One stream's partition of the views table `table`. @@ -303,4 +307,19 @@ impl Table for DynamoDbTable { ) .await } + + async fn query_typed(&self, item_type: &str) -> Result, DdbError> { + self.query_all( + Some(TYPED_INDEX), + "#t = :t", + &[("#t", TYPED)], + &[( + ":t", + AttributeValue::S(typed_partition(&self.stream, item_type)), + )], + false, + None, + ) + .await + } } diff --git a/crates/fold-ddb/src/fold.rs b/crates/fold-ddb/src/fold.rs index 6d0c283..9770de4 100644 --- a/crates/fold-ddb/src/fold.rs +++ b/crates/fold-ddb/src/fold.rs @@ -21,7 +21,11 @@ //! bumps it, and every stream folded before is no longer current and is folded again the same way. //! Revision 1 (6j6v.t1ym) adds the `.ladj#`/`.tadj#` entries; a stream folded before it has none, //! so its watermark names no index revision at all and the next run refolds it. Revision 2 -//! (6j6v.15ed) adds the `.cadj#` entries, a ticket's outgoing `contributes_to` edges. +//! (6j6v.15ed) adds the `.cadj#` entries, a ticket's outgoing `contributes_to` edges. Revision 3 +//! (6j6v.1c5s) makes the `.adj#` entries self-describing — ends, kind, coordinate, presence — for +//! the derived ranking keys ([`crate::records::with_ranking`]), and sets the `typed` index +//! attribute on every live item row (6j6v.xkr7, [`crate::records::list_board`]). Both ship in one +//! release: no stream was ever folded by revision 3 without the attribute. //! //! # What the server cannot see //! @@ -32,7 +36,9 @@ //! coordinate. use crate::board; -use crate::layout::{Layout, LayoutError, MAX_ITEM_BYTES, MAX_KEY_BYTES, WATERMARK_KEY}; +use crate::layout::{ + Layout, LayoutError, MAX_ITEM_BYTES, MAX_KEY_BYTES, MAX_PARTITION_BYTES, WATERMARK_KEY, +}; use crate::table::{Outcome, Table}; use crate::write::{plan, Cond, Write}; use nxs_foundation::change::{Cell, Change, Wins}; @@ -46,7 +52,9 @@ pub const INDEX_DOMAIN: &str = "index"; /// The revision of those rules. 1: the entries under a ticket for its label and thread link adds /// (6j6v.t1ym). 2: the entries under a ticket for its outgoing `contributes_to` edges (6j6v.15ed). -pub const INDEX_REVISION: i64 = 2; +/// 3: the adjacency entries copy their edge's ends, kind and coordinate and say whether it is +/// removed (6j6v.1c5s), and every live item row carries its `typed` index partition (6j6v.xkr7). +pub const INDEX_REVISION: i64 = 3; /// The platform's reducers — the board's, the chat's and the memory's. pub fn platform_reducers() -> Vec> { @@ -158,7 +166,8 @@ impl Folder { /// Fold one op: carry out its changes, then keep the board's indexes. /// - /// An op the table cannot hold — a key past `MAX_KEY_BYTES`, a row past `MAX_ITEM_BYTES` — is + /// An op the table cannot hold — a key past `MAX_KEY_BYTES`, a type whose `typed` index + /// partition is past `MAX_PARTITION_BYTES`, a row past `MAX_ITEM_BYTES` — is /// [`Folded::Refused`] before it writes anything, rather than failing the batch for ever: the /// relay keeps it, as it keeps any op a server does not fold. A row that only outgrows the limit /// through earlier ops refuses the one write that would break it ([`Outcome::TooLarge`]); the @@ -193,6 +202,16 @@ impl Folder { ))); } } + if let Some(p) = board::typed_partitions(table.stream(), &changes) + .into_iter() + .find(|p| p.len() > MAX_PARTITION_BYTES) + { + return Ok(refuse(format!( + "a type whose `typed` index partition has {} bytes, past the {MAX_PARTITION_BYTES} \ + a partition key may have", + p.len() + ))); + } if let Some(w) = writes.iter().find(|w| w.bytes() > MAX_ITEM_BYTES) { return Ok(refuse(format!( "a row of {} bytes, past the {MAX_ITEM_BYTES} an item may have", diff --git a/crates/fold-ddb/src/layout.rs b/crates/fold-ddb/src/layout.rs index ffd4943..73475e6 100644 --- a/crates/fold-ddb/src/layout.rs +++ b/crates/fold-ddb/src/layout.rs @@ -26,18 +26,26 @@ pub const ACTIVE: &str = "nxf_active"; pub const DATED: &str = "nxf_dated"; /// The sort key of the `dated` index: the instant the ticket entered its lane. pub const DATED_AT: &str = "nxf_dated_at"; +/// The partition key of the `typed` index (6j6v.xkr7): `#` ([`typed_partition`]), +/// present only on a live item row — not deleted, archived or not — that has a type. +pub const TYPED: &str = "nxf_typed"; /// The `active` index. pub const ACTIVE_INDEX: &str = "active"; /// The `dated` index. pub const DATED_INDEX: &str = "dated"; +/// The `typed` index (6j6v.xkr7). +pub const TYPED_INDEX: &str = "typed"; /// The longest sort key — and `dated` index sort key — DynamoDB stores, in bytes. pub const MAX_KEY_BYTES: usize = 1024; +/// The longest partition key value DynamoDB stores — on the table or on an index — in bytes. Only +/// the `typed` index's can come near it: it carries an item's type ([`typed_partition`]). +pub const MAX_PARTITION_BYTES: usize = 2048; /// The largest item DynamoDB stores, in bytes, counting attribute names and values. pub const MAX_ITEM_BYTES: usize = 400 * 1024; /// Attribute names no view column may take: the table's own keys and the index attributes. -pub const RESERVED: [&str; 5] = [PK, SK, ACTIVE, DATED, DATED_AT]; +pub const RESERVED: [&str; 6] = [PK, SK, ACTIVE, DATED, DATED_AT, TYPED]; /// The fold's own bookkeeping, outside every view table: a view table's name is an identifier and /// cannot start with `.`. @@ -69,6 +77,13 @@ pub fn row_key(table: &str, key: &[(String, Cell)]) -> Result#`, the +/// type with `%` and `#` escaped as a key value is, so two streams' types can never share a +/// partition (`a` + `b#c` and `a#b` + `c`). +pub fn typed_partition(stream: &str, item_type: &str) -> String { + format!("{stream}#{}", escape(item_type)) +} + /// The prefix every row of `table` starts with. pub fn table_prefix(table: &str) -> String { format!("{table}#") diff --git a/crates/fold-ddb/src/lib.rs b/crates/fold-ddb/src/lib.rs index 868b37b..05e279d 100644 --- a/crates/fold-ddb/src/lib.rs +++ b/crates/fold-ddb/src/lib.rs @@ -1,7 +1,8 @@ //! Folding on a server (6j6v.vvw6): the reducers' [described changes](nxs_foundation::change) //! carried out as conditional writes against one table per deployment, the board's `active` and //! `dated` indexes kept beside the rows, next/blocked derived by the same library a local replica -//! uses, and snapshots that are tables plus watermark, without the log. +//! uses, and snapshots that are tables plus watermark, without the log; a type's live items listed +//! by a third index, `typed`, for a ranked hosted `list`. //! //! The pieces, in the order a fold run uses them: [`fold::Folder`] maps each op through the //! platform's reducers and [`write::plan`] turns every change into a [`write::Write`]; @@ -22,23 +23,42 @@ //! | sort key | `sk` | S | `#`, `.adj#`/`.ladj#`/`.tadj##`, `.meta#watermark` | //! | GSI `active` | `nxf_active` / `sk` | S/S | sparse: active tickets and the present dep/parent edges touching one | //! | GSI `dated` | `nxf_dated` / `nxf_dated_at` | S/S | sparse: `#closed` and `#archived`, by `␟` | +//! | GSI `typed` | `nxf_typed` / `sk` | S/S | sparse: `#` on every live item row (not deleted; archived included) | //! -//! Both indexes project ALL attributes, so a lane is one Query and no follow-up reads. The lanes +//! The `.adj##` entries are plain rows in the stream's partition, on no index. Since +//! index revision 3 (6j6v.1c5s) each also carries its edge's `from_id`, `to_id`, `kind`, `lamport`, +//! `site` and a `removed` flag — attributes, not keys. +//! +//! Every index projects ALL attributes, so a lane is one Query and no follow-up reads. +//! +//! **The `typed` index (6j6v.xkr7)** is the one change to this contract since `active` and `dated` +//! (docs/specs/derived-ranking-keys.md §6, owner decision §10.4): one Query lists the live items of +//! a type, the `list` scope a hosted `list` and the derived ranking keys rank over +//! ([`records::list_board`]). Its partition key `nxf_typed` is `#`, the type escaped +//! as a key value is (`%` → `%25`, `#` → `%23`; [`layout::typed_partition`]); its sort key is the +//! table's `sk`. The fold sets it on every item row whose type is set and which is not deleted — +//! closed and archived ones included — and removes it on a delete, written after the row like the +//! other index attributes ([`board`], "`typed`"). The deploying stack adds the GSI FIRST: a fold +//! against a table without it still writes the attribute, which costs nothing and breaks nothing; +//! only the read — a `list` through [`records::list_board`], a `next` whose keys read a +//! `[ranking.list]` rank — needs the index. A type whose partition would pass DynamoDB's 2,048 +//! bytes for an index key is refused at fold time ([`layout::MAX_PARTITION_BYTES`]). The lanes //! need nothing else; the RECORDS a lane is shown with carry joins from other tables, which //! [`records`] reads with a fixed number of requests per active ticket. Billing mode //! is the deployment's call. No local secondary index: without one, DynamoDB sets no limit on the //! size of a stream's partition (the 10 GB item-collection limit applies only to tables with an LSI). //! What does bound a stream is the throughput of ONE partition key: a stream's `active` and `dated` -//! entries each share one index partition — a write-heavy stream is limited to that partition's -//! throughput (1,000 writes per second). +//! entries each share one index partition, its items of one type one `typed` partition — a +//! write-heavy stream is limited to that partition's throughput (1,000 writes per second). //! //! Every row of view table `t` carries its columns as attributes of the same name ([`layout`]); a //! NULL cell is DynamoDB's NULL, an integer a number, text a string. //! //! # Freshness //! -//! A row read (`GetItem`, a Query on the table) is strongly consistent. The two index reads — -//! `board::select` and `board::dated` — are eventually consistent, as every DynamoDB GSI read is: +//! A row read (`GetItem`, a Query on the table) is strongly consistent. The index reads — +//! `board::select`, `board::dated` and the `typed` Query of [`records::list_board`] — are eventually +//! consistent, as every DynamoDB GSI read is: //! right after a fold they may answer the state before it, for as long as the index takes to catch //! up (typically well under a second). The lanes are the local lanes of THAT state. A reader that //! must see its own write reads the ticket's row instead. @@ -61,7 +81,14 @@ //! one write. A label or thread link add, or its remove, adds two `GetItem`s and at most one write //! for the entry under the ticket (`board`, "Who owns a label or a thread link"). Reading the //! active tickets' records costs `4n + 1` requests for `n` tickets, plus one per parent outside -//! the selection, at most [`records::READ_CONCURRENCY`] tickets at a time ([`records`]). No request +//! the selection, at most [`records::READ_CONCURRENCY`] tickets at a time ([`records`]). Folding an +//! edge remove reads the add and the remove and rewrites the entry under each end. The graph the +//! derived ranking keys read ([`records::with_ranking`]) costs one Query per active ticket and, per +//! inactive neighbour, one `GetItem` plus up to three Queries. A hosted `list` +//! ([`records::list_board`]) costs one `typed` Query per listed type and `4n + 1 + p` requests for +//! `n` listed items, as the records read. The `typed` index costs a fold at most one write per item +//! op, and only when the op gains, changes or loses the item's partition — a create, a retype, a +//! delete — in the same write as the item's other index attributes when they move too. No request //! is a Scan: every read names its partition key, the board's through an index. The //! `dynamodb-local` CI job counts the requests of a fold and of a records read, and proves it. //! diff --git a/crates/fold-ddb/src/mem.rs b/crates/fold-ddb/src/mem.rs index 09178aa..155b9b1 100644 --- a/crates/fold-ddb/src/mem.rs +++ b/crates/fold-ddb/src/mem.rs @@ -7,7 +7,7 @@ //! It counts the requests it answers ([`Requests`]), so the cost of a fold — the flag fan-out above //! all — is measured on the same code path the DynamoDB table runs. -use crate::layout::{ACTIVE, DATED, DATED_AT, MAX_ITEM_BYTES, PK, SK}; +use crate::layout::{typed_partition, ACTIVE, DATED, DATED_AT, MAX_ITEM_BYTES, PK, SK, TYPED}; use crate::table::{Dated, Outcome, Row, Table}; use crate::write::{attribute_bytes, beats, Cond, Write}; use nxs_foundation::change::Cell; @@ -176,6 +176,20 @@ impl Table for MemTable { rows.truncate(limit.unwrap_or(usize::MAX)); Ok(rows) } + + async fn query_typed(&self, item_type: &str) -> Result, Infallible> { + self.count(|r| r.index_queries += 1); + let partition = Cell::Text(typed_partition(&self.stream, item_type)); + // The rows are kept by sort key: the index's own order. + Ok(self + .rows + .lock() + .unwrap() + .values() + .filter(|r| r.get(TYPED) == Some(&partition)) + .cloned() + .collect()) + } } /// Run a future that never waits — every [`MemTable`] future — to completion on this thread. diff --git a/crates/fold-ddb/src/records.rs b/crates/fold-ddb/src/records.rs index cb930d8..6560c95 100644 --- a/crates/fold-ddb/src/records.rs +++ b/crates/fold-ddb/src/records.rs @@ -34,6 +34,20 @@ //! a time, so a board's latency is about `n / READ_CONCURRENCY` round trips, and no board sends more //! than that many requests at once. //! +//! One more is read only for a plugin that ranks `next` by derived ranking keys: +//! [`with_ranking`] fills the board's [`RankingGraph`] — the edges around the active tickets and +//! their live neighbours — from the adjacency entries (index revision 3, 6j6v.1c5s), and the live +//! items of a type whose `[ranking.list]` rank a key reads from the `typed` index (6j6v.xkr7). Its +//! cost is in its own doc. +//! +//! # A hosted `list` +//! +//! [`list_board`] reads what the facade's store-free `list` ranks and projects (6j6v.xkr7): the +//! live, unarchived items of the listed types off the `typed` index, with the joins a `list` +//! record and its filters carry, `next`'s board when a key reads a rank `next` gives, and the +//! graph. The server asks [`read::list_reads`](nexus_flow_facade::read::list_reads) what to read +//! for its query; its cost is in its own doc. +//! //! # Two points in time //! //! The selection is an index read, eventually consistent; every join is a row read, strongly @@ -47,23 +61,29 @@ //! # A stream folded by older rules //! //! The entries under a ticket came with index revision 1, the `.cadj#` entries with revision 2 -//! (6j6v.15ed). A stream whose watermark names an older -//! one — or none, folded before revisions were recorded for the index — has no entries, and its -//! records would come back without labels or conversations. [`active_board`] refuses such a stream -//! ([`RecordsError::NotCurrent`]); the fold run refolds it ([`Folder::is_current`]), after which it -//! reads normally. +//! (6j6v.15ed), the self-describing `.adj#` entries and the `typed` index attribute with revision 3 +//! (6j6v.1c5s, 6j6v.xkr7). A stream whose watermark names an older one — or none, folded before +//! revisions were recorded for the index — lacks them, and its records would come back without +//! labels or conversations, its list empty. [`active_board`] and [`list_board`] refuse such a +//! stream ([`RecordsError::NotCurrent`]); the fold run refolds it ([`Folder::is_current`]), after +//! which it reads normally. //! //! [`Folder::is_current`]: crate::fold::Folder::is_current -use crate::board::{int, item_key, Selection, REMOVED}; +use crate::board::{int, is_active, item_key, select, Selection, REMOVED}; use crate::fold::{Folder, INDEX_DOMAIN, INDEX_REVISION}; use crate::layout::{ - contributes_adjacency_prefix, label_adjacency_prefix, link_adjacency_prefix, row_key, + adjacency_prefix, contributes_adjacency_prefix, label_adjacency_prefix, link_adjacency_prefix, + row_key, table_prefix, }; use crate::table::{text, Row, Table}; use futures_util::stream::{self, StreamExt, TryStreamExt}; -use nexus_flow_core::model::{ItemRow, LinkWeight}; -use nexus_flow_facade::read::{ActiveBoard, ActiveTicket, ParentRef}; +use nexus_flow_core::graph::Edge; +use nexus_flow_core::model::{EdgeKind, ItemRow, LinkWeight}; +use nexus_flow_facade::read::{ + ActiveBoard, ActiveTicket, ListBoard, ListItem, ListReads, Neighbour, ParentRef, RankingGraph, + RankingReads, +}; use nxs_foundation::change::Cell; use std::collections::{BTreeMap, BTreeSet}; @@ -110,12 +130,7 @@ pub async fn active_board( table: &T, selection: Selection, ) -> Result> { - if let Some(mark) = Folder::watermark(table).await? { - let found = mark.revisions.get(INDEX_DOMAIN).copied(); - if found != Some(INDEX_REVISION) { - return Err(RecordsError::NotCurrent { found }); - } - } + check_current(table).await?; let Selection { items, board, @@ -206,6 +221,462 @@ async fn contributes_to(table: &T, id: &str) -> Result, T: Ok(targets.into_iter().collect()) } +/// One present edge as its adjacency entry tells it (index revision 3): the add's ends, kind and +/// coordinate (`-1` where the add has none, as `present_parent` reads it). +#[derive(Debug, Clone)] +struct EdgeEntry { + from: String, + to: String, + kind: EdgeKind, + lamport: i64, + site: i64, +} + +/// The present edges touching ticket `id`, by tag: one Query on its `.adj#` prefix. +async fn present_edges( + table: &T, + id: &str, +) -> Result, T::Error> { + Ok(present_entries(table, &adjacency_prefix(id)) + .await? + .iter() + .filter_map(|e| { + let entry = EdgeEntry { + from: text(e, "from_id")?.to_string(), + to: text(e, "to_id")?.to_string(), + kind: text(e, "kind").and_then(EdgeKind::parse)?, + lamport: int(e, "lamport").unwrap_or(-1), + site: int(e, "site").unwrap_or(-1), + }; + Some((text(e, "tag")?.to_string(), entry)) + }) + .collect()) +} + +/// The current parent of `id` among the present edges touching it — `present_parent`'s pick: of its +/// present outgoing `parent` edges, the one with the highest `(lamport, site, to_id)`. +fn current_parent(id: &str, edges: &BTreeMap) -> Option { + edges + .values() + .filter(|e| e.kind == EdgeKind::Parent && e.from == id) + .max_by(|a, b| (a.lamport, a.site, &a.to).cmp(&(b.lamport, b.site, &b.to))) + .map(|e| e.to.clone()) +} + +/// `board` with its [`RankingGraph`] for `reads` (6j6v.1c5s): the graph the derived ranking keys of +/// `[ranking.next]` read, from the folded tables — what [`read::with_ranking`] reads from a store, +/// edge for edge and neighbour for neighbour. A server calls it when +/// [`read::next_ranking_reads`] names reads for its plugin, so a plugin without derived `next` keys +/// pays nothing for it. The stream must be current, as for [`active_board`]: the entries read here +/// came with index revision 3. +/// +/// | read | per | +/// |-----------------------------------------------|-------------------------------------------| +/// | one Query on `.adj##` | active ticket | +/// | one Query on the `typed` index | type in `reads.list_types` (6j6v.xkr7) | +/// | one Query on `.adj##` | live, unarchived item of such a type that is not active | +/// | one `GetItem` of its row | other end of such an edge that is not a subject | +/// | one Query on `.adj##` (its parent) | such end that is live | +/// | one Query on `.ladj##`, `custom_fields##` | such end and listed item, when `reads` asks for labels, custom values | +/// +/// So `n` active tickets with `p` distinct inactive neighbours, `q` of them live, cost `n + q` +/// Queries and `p` `GetItem`s — `n + 3q` Queries with labels and custom values — at most +/// [`READ_CONCURRENCY`] items at a time, and nothing grows with an item's history: an entry says +/// whether its add is removed. Every edge touching a subject is read once, from one end's entries, +/// whatever the plugin's keys — the edge pass the store path runs once per query. A `next` key +/// that reads the rank of a type only `[ranking.list]` orders adds, per such type, one index Query +/// and its live items as subjects ([`RankingReads::list_types`]). +/// +/// [`read::with_ranking`]: nexus_flow_facade::read::with_ranking +/// [`read::next_ranking_reads`]: nexus_flow_facade::read::next_ranking_reads +pub async fn with_ranking( + table: &T, + mut board: ActiveBoard, + reads: &RankingReads, +) -> Result> { + let ids: Vec = board.tickets.keys().cloned().collect(); + let held: BTreeMap = stream::iter(ids) + .map(|id| async move { + let edges = present_edges(table, &id).await?; + Ok::<_, T::Error>((id, edges)) + }) + .buffered(READ_CONCURRENCY) + .try_collect::>() + .await? + .into_iter() + .map(|(id, edges)| { + let parent = board + .tickets + .get(&id) + .and_then(|t| t.item.belongs_to.clone()); + (id, Held { parent, edges }) + }) + .collect(); + board.ranking = Some(ranking_graph(table, reads, held).await?); + Ok(board) +} + +/// A row the board holds itself — an active ticket, a listed item — as the graph reads it: its +/// current parent and the present edges touching it. +struct Held { + parent: Option, + edges: BTreeMap, +} + +/// The live, unarchived items of `item_type` on the `typed` index — the type's `list` scope — +/// asked of the row again, since the index may lag it. In id order. +async fn typed_items(table: &T, item_type: &str) -> Result, T::Error> { + Ok(table + .query_typed(item_type) + .await? + .into_iter() + .filter(|r| { + crate::board::typed(r) == Some(item_type) + && text(r, "archived").is_none() + && text(r, "id").is_some() + }) + .collect()) +} + +/// One live row the graph holds, read in full: its row with its current parent, its labels and +/// custom values when `reads` asks, and its present edges. +async fn neighbour( + table: &T, + id: &str, + row: &Row, + reads: &RankingReads, +) -> Result<(Neighbour, BTreeMap), T::Error> { + let edges = present_edges(table, id).await?; + let mut n = Neighbour::new(item_row(row, current_parent(id, &edges))); + if reads.labels { + n.labels = labels(table, id).await?; + } + if reads.custom { + n.custom = custom(table, id).await?; + } + Ok((n, edges)) +} + +/// The [`RankingGraph`] for `reads` around the rows a board holds itself (`held`, each a subject, +/// with its edges already read) — what [`with_ranking`] and [`list_board`] put on their boards. +async fn ranking_graph( + table: &T, + reads: &RankingReads, + held: BTreeMap, +) -> Result { + // The listed items of `reads.list_types` the board does not hold: subjects too. + let mut listed: BTreeSet = BTreeSet::new(); + let mut scope_rows: Vec<(String, Row)> = Vec::new(); + for ty in &reads.list_types { + for row in typed_items(table, ty).await? { + let id = text(&row, "id").unwrap_or_default().to_string(); + listed.insert(id.clone()); + if !held.contains_key(&id) { + scope_rows.push((id, row)); + } + } + } + let scope: BTreeMap)> = + stream::iter(scope_rows) + .map(|(id, row)| async move { + let read = neighbour(table, &id, &row, reads).await?; + Ok::<_, T::Error>((id, read)) + }) + .buffered(READ_CONCURRENCY) + .try_collect() + .await?; + let subject = |id: &str| held.contains_key(id) || scope.contains_key(id); + + // Every present edge of a kind read that touches a subject, once per tag. + let mut touching: BTreeMap = BTreeMap::new(); + for edges in held + .values() + .map(|h| &h.edges) + .chain(scope.values().map(|(_, e)| e)) + { + touching.extend( + edges + .iter() + .filter(|(_, e)| reads.kinds.contains(&e.kind)) + .map(|(tag, e)| (tag.clone(), e.clone())), + ); + } + + // The other ends: their rows, and for the live ones their current parent and the joins asked + // for. + let others: BTreeSet = touching + .values() + .flat_map(|e| [&e.from, &e.to]) + .filter(|id| !subject(id)) + .cloned() + .collect(); + let rows: Vec<(String, Option)> = stream::iter(others) + .map(|id| async move { + let row = table.get(&item_key(&id)).await?; + Ok::<_, T::Error>((id, row)) + }) + .buffered(READ_CONCURRENCY) + .try_collect() + .await?; + let live = rows + .into_iter() + .filter_map(|(id, row)| row.map(|r| (id, r))) + .filter(|(_, r)| text(r, "deleted") != Some("1")); + let ends: BTreeMap = stream::iter(live) + .map(|(id, row)| async move { + let (n, _) = neighbour(table, &id, &row, reads).await?; + Ok::<_, T::Error>((id, n)) + }) + .buffered(READ_CONCURRENCY) + .try_collect() + .await?; + + // The edges between live ends, a `parent` edge only from a child to its current parent. + let parent_of = |id: &str| -> Option<&str> { + match held.get(id) { + Some(h) => h.parent.as_deref(), + None => scope + .get(id) + .map(|(n, _)| n) + .or_else(|| ends.get(id)) + .and_then(|n| n.item.belongs_to.as_deref()), + } + }; + let live_end = |id: &str| subject(id) || ends.contains_key(id); + let edges: BTreeSet = touching + .into_values() + .filter(|e| live_end(&e.from) && live_end(&e.to)) + .filter(|e| e.kind != EdgeKind::Parent || parent_of(&e.from) == Some(e.to.as_str())) + .map(|e| Edge { + from: e.from, + to: e.to, + kind: e.kind, + }) + .collect(); + // A child whose current parent is elsewhere was read for its parent edge only, which is not in + // the graph: it is no neighbour. A listed item always is one. + let edge_ends: BTreeSet<&str> = edges + .iter() + .flat_map(|e| [e.from.as_str(), e.to.as_str()]) + .collect(); + let mut neighbours: BTreeMap = ends + .into_iter() + .filter(|(id, _)| edge_ends.contains(id.as_str())) + .collect(); + neighbours.extend(scope.into_iter().map(|(id, (n, _))| (id, n))); + let mut graph = RankingGraph::new(reads.clone()); + graph.edges = edges; + graph.neighbours = neighbours; + graph.listed = listed; + Ok(graph) +} + +/// The [`ListBoard`] for `reads` from the folded tables (6j6v.xkr7) — what [`read::list_board`] +/// reads from a store, item for item — for the facade's store-free `list` +/// ([`read::list_board_page`], [`read::list_board_value`]). A server takes `reads` from +/// [`read::list_reads`] for its query. The stream must be current, as for [`active_board`]: the +/// `typed` index and the self-describing `.adj#` entries came with index revision 3. +/// +/// | read | per | +/// |---------------------------------------------------------|----------------------------------| +/// | one `GetItem` of `.meta#watermark` | board | +/// | one Query on the `typed` index | listed type | +/// | one Query on the partition's `items#` rows | board without a type filter | +/// | one Query each on `.adj#`, `.ladj#`, `custom_fields#`; one `GetItem` of `item_timestamps` | listed item | +/// | one `GetItem` | parent of an open listed item outside the list | +/// | one Query on the `active` index, and per active ticket one Query each on `.ladj#` / `custom_fields#` when the graph reads them | board whose keys read a `next` rank | +/// | [`with_ranking`]'s reads around the listed items and active tickets | board with derived keys | +/// +/// So `n` listed items cost `4n + 1 + p` requests and one Query per listed type, as +/// [`active_board`] costs per active ticket; the `.adj#` Query gives each item's parent, its +/// `contributes_to` targets and its parents for `parent_closed_reason` at once, and the graph reuses +/// it. Archived and deleted items are dropped before any join: `list` returns neither (archived +/// ones are on the index, live as they are, and filtered here as `list` filters them locally). +/// +/// A query without a type filter lists every live item, so it reads the partition's `items#` rows +/// — strongly consistent, every item row of the stream, deleted ones included — instead of an +/// index. The index lists one type; a query that names its types reads only those. +/// +/// [`read::list_board`]: nexus_flow_facade::read::list_board +/// [`read::list_board_page`]: nexus_flow_facade::read::list_board_page +/// [`read::list_board_value`]: nexus_flow_facade::read::list_board_value +/// [`read::list_reads`]: nexus_flow_facade::read::list_reads +pub async fn list_board( + table: &T, + reads: &ListReads, +) -> Result> { + check_current(table).await?; + let rows: Vec = match &reads.types { + Some(types) => { + let mut rows = Vec::new(); + for ty in types { + rows.extend(typed_items(table, ty).await?); + } + rows + } + None => table + .query_prefix(&table_prefix("items")) + .await? + .into_iter() + .filter(|r| { + text(r, "deleted") != Some("1") + && text(r, "archived").is_none() + && text(r, "id").is_some() + }) + .collect(), + }; + let read: Vec<(Row, BTreeMap, ListItem)> = stream::iter(rows) + .map(|row| async move { + let id = text(&row, "id").unwrap_or_default().to_string(); + let edges = present_edges(table, &id).await?; + let (created_at, updated_at) = timestamps(table, &id).await?; + let mut i = ListItem::new(item_row(&row, current_parent(&id, &edges))); + i.labels = labels(table, &id).await?; + i.custom = custom(table, &id).await?; + i.created_at = created_at; + i.updated_at = updated_at; + let targets: BTreeSet = edges + .values() + .filter(|e| e.kind == EdgeKind::ContributesTo && e.from == id) + .map(|e| e.to.clone()) + .collect(); + i.contributes_to = targets.into_iter().collect(); + Ok::<_, T::Error>((row, edges, i)) + }) + .buffered(READ_CONCURRENCY) + .try_collect() + .await?; + + // `parent_closed_reason`: an OPEN item with parents, none of them active, lists them all. A + // parent outside the list is read once, however many children it has. + let parents_of = |id: &str, edges: &BTreeMap| -> BTreeSet { + edges + .values() + .filter(|e| e.kind == EdgeKind::Parent && e.from == id) + .map(|e| e.to.clone()) + .collect() + }; + let listed_rows: BTreeMap<&str, &Row> = read + .iter() + .map(|(row, _, i)| (i.item.id.as_str(), row)) + .collect(); + let outside: BTreeSet = read + .iter() + .filter(|(_, _, i)| i.item.status.as_deref() == Some("open")) + .flat_map(|(_, edges, i)| parents_of(&i.item.id, edges)) + .filter(|p| !listed_rows.contains_key(p.as_str())) + .collect(); + let outside: BTreeMap> = stream::iter(outside) + .map(|pid| async move { + let row = table.get(&item_key(&pid)).await?; + Ok::<_, T::Error>((pid, row)) + }) + .buffered(READ_CONCURRENCY) + .try_collect() + .await?; + let parent_row = |pid: &str| -> Option<&Row> { + listed_rows + .get(pid) + .copied() + .or_else(|| outside.get(pid).and_then(Option::as_ref)) + }; + let parent_closed: Vec)>> = read + .iter() + .map(|(_, edges, i)| { + if i.item.status.as_deref() != Some("open") { + return Vec::new(); + } + let mut closed = Vec::new(); + for pid in parents_of(&i.item.id, edges) { + match parent_row(&pid) { + Some(r) if is_active(r) => return Vec::new(), + Some(r) => closed.push(( + text(r, "id").unwrap_or(&pid).to_string(), + text(r, "closing_comment").map(str::to_string), + )), + None => closed.push((pid, None)), + } + } + closed.sort_by(|a, b| a.0.cmp(&b.0)); + closed + }) + .collect(); + let mut items = Vec::with_capacity(read.len()); + let mut held: BTreeMap = BTreeMap::new(); + for ((_, edges, mut i), closed) in read.into_iter().zip(parent_closed) { + i.parent_closed = closed; + held.insert( + i.item.id.clone(), + Held { + parent: i.item.belongs_to.clone(), + edges, + }, + ); + items.push(i); + } + let mut board = ListBoard::new(reads.types.clone(), items); + + let graph_reads = reads.ranking.clone().unwrap_or_default(); + if reads.next { + let Selection { + items: rows, + board: lanes, + parents, + } = select(table).await?; + let tickets: Vec = stream::iter(rows) + .map(|(id, row)| { + let parent = parents.get(&id).cloned(); + let graph_reads = &graph_reads; + async move { + let mut t = ActiveTicket::new(item_row(&row, parent)); + if graph_reads.labels { + t.labels = labels(table, &id).await?; + } + if graph_reads.custom { + t.custom = custom(table, &id).await?; + } + Ok::<_, T::Error>(t) + } + }) + .buffered(READ_CONCURRENCY) + .try_collect() + .await?; + board.active = Some(ActiveBoard::new(lanes, tickets)); + } + if let Some(r) = &reads.ranking { + if let Some(active) = &board.active { + let missing: Vec<(String, Option)> = active + .tickets + .iter() + .filter(|(id, _)| !held.contains_key(*id)) + .map(|(id, t)| (id.clone(), t.item.belongs_to.clone())) + .collect(); + let more: Vec<(String, Held)> = stream::iter(missing) + .map(|(id, parent)| async move { + let edges = present_edges(table, &id).await?; + Ok::<_, T::Error>((id, Held { parent, edges })) + }) + .buffered(READ_CONCURRENCY) + .try_collect() + .await?; + held.extend(more); + } + board.ranking = Some(ranking_graph(table, r, held).await?); + } + Ok(board) +} + +/// Refuse a stream folded by older index rules than this build reads. +async fn check_current(table: &T) -> Result<(), RecordsError> { + if let Some(mark) = Folder::watermark(table).await? { + let found = mark.revisions.get(INDEX_DOMAIN).copied(); + if found != Some(INDEX_REVISION) { + return Err(RecordsError::NotCurrent { found }); + } + } + Ok(()) +} + /// An `items` row as the store reads it, with the ticket's `belongs_to` (the `present_parent` join). pub(crate) fn item_row(row: &Row, belongs_to: Option) -> ItemRow { let cell = |c: &str| text(row, c).map(str::to_string); diff --git a/crates/fold-ddb/src/table.rs b/crates/fold-ddb/src/table.rs index c382c1e..51cd92b 100644 --- a/crates/fold-ddb/src/table.rs +++ b/crates/fold-ddb/src/table.rs @@ -1,6 +1,6 @@ -//! The five requests a server fold makes of its table, as a trait: [`MemTable`](crate::mem::MemTable) +//! The requests a server fold makes of its table, as a trait: [`MemTable`](crate::mem::MemTable) //! answers them in memory for the tests, `ddb::DynamoDbTable` (feature `dynamodb`) answers them with DynamoDB. -//! None of them is a Scan — every read names its partition, and the two board reads name an index. +//! None of them is a Scan — every read names its partition, and the three index reads name an index. use crate::write::Write; use nxs_foundation::change::Cell; @@ -71,6 +71,13 @@ pub trait Table: Sync { lane: Dated, limit: Option, ) -> impl Future, Self::Error>> + Send; + + /// Every row on the `typed` index under `item_type` (6j6v.xkr7): the live items of that type, + /// archived ones included, in sort-key (id) order. Eventually consistent, as every index read. + fn query_typed( + &self, + item_type: &str, + ) -> impl Future, Self::Error>> + Send; } /// A text cell of `row`, `None` when it is absent or NULL. diff --git a/crates/fold-ddb/tests/common/mod.rs b/crates/fold-ddb/tests/common/mod.rs index eeffa36..8d4274c 100644 --- a/crates/fold-ddb/tests/common/mod.rs +++ b/crates/fold-ddb/tests/common/mod.rs @@ -25,6 +25,10 @@ impl Lcg { } } +/// The item types of a generated board: items are created as one of the first two and retyped +/// among all three, so the `typed` index sees a type gained, lost and changed (6j6v.xkr7). +pub const TYPES: [&str; 3] = ["task", "epic", "note"]; + pub fn id(i: u64) -> String { format!("ab12.{i:04}") } @@ -38,6 +42,11 @@ fn write(s: &mut Store, r: &mut Lcg, n: u64, steps: usize) { step % 24 )); let (a, b) = (id(r.next(n)), id(r.next(n + 1))); + // Now and then a retype (6j6v.xkr7), on the step alone so the random stream — and with it + // every other op of a seed — stays as it was. + if step % 9 == 4 { + s.set_field(&a, "type", Some(TYPES[step % TYPES.len()].into()), "u"); + } match r.next(16) { 0 => s.create_item(&a, "task", "T", "u"), 1 => s.set_field(&a, "status", Some("in_progress".into()), "u"), @@ -149,7 +158,7 @@ pub fn merged(seed: u64) -> Store { let mut a = Store::open_in_memory(1); let mut b = Store::open_in_memory(2); for i in 0..n { - a.create_item(&id(i), "task", "T", "u"); + a.create_item(&id(i), TYPES[(i % 2) as usize], "T", "u"); } b.apply(&a.export()); let steps = 10 + r.next(40) as usize; diff --git a/crates/fold-ddb/tests/derived_differential.rs b/crates/fold-ddb/tests/derived_differential.rs new file mode 100644 index 0000000..cd5b228 --- /dev/null +++ b/crates/fold-ddb/tests/derived_differential.rs @@ -0,0 +1,483 @@ +//! A server ranks `next` by derived ranking keys as a replica does (6j6v.1c5s, +//! docs/specs/derived-ranking-keys.md §6 and §9 "differential"). +//! +//! Generated boards of three types — goals, projects, actions — written by two replicas apart and +//! then merged: every status, archived and deleted items, priorities, the goals' `order` custom +//! field, `waiting` labels, and `parent`, `contributes_to`, `mentions` and `dep` edges with removes. +//! Two replicas parent the same child apart, so a child can have two present parent edges and only +//! `present_parent`'s pick may count. Each board is folded into a [`MemTable`] in a shuffled order +//! with duplicates, as the relay hands it over. +//! +//! Then three paths rank `next` under a plugin whose orders hold every derived key kind, each +//! reading closed, archived and deleted neighbours, down a three-type chain (action → project → +//! goal): +//! +//! 1. the SQL path: `read::next` over the store; +//! 2. the Board path: `read::next_active` over the store's own [`read::ActiveBoard`] and +//! [`read::RankingGraph`] (`read::active_board` + `read::with_ranking`); +//! 3. the fold-ddb path: `read::next_active` over the folded tables' (`board::select` + +//! `records::active_board` + `records::with_ranking`). +//! +//! All three must give the same rows in the same order and the same JSON, and the two inputs must +//! be equal, edge for edge and neighbour for neighbour. Folding the same log in another order must +//! read the same graph (determinism). + +use nexus_flow_core::model::{EdgeKind, ItemRow, MergeStrategy}; +use nexus_flow_core::store::Store; +use nexus_flow_facade::plugin::{self, PluginConfig, PluginRegistration}; +use nexus_flow_facade::read; +use nxs_fold_ddb::fold::{Folder, INDEX_DOMAIN}; +use nxs_fold_ddb::layout::WATERMARK_KEY; +use nxs_fold_ddb::mem::{ready, MemTable}; +use nxs_fold_ddb::table::Table; +use nxs_fold_ddb::write::{Cond, Write}; +use nxs_fold_ddb::{board, records}; +use nxs_foundation::change::Cell; +use std::collections::{BTreeMap, BTreeSet}; + +mod common; +use common::{delivered, Lcg, NOW}; + +/// Every derived key kind, each able to read a neighbour that is not active: a goal's order and +/// rank from a closed or archived goal, counts without a status filter, a `dep` blocker's status +/// (only a closed blocker leaves its dependent in `next`), the `belongs_to` of a mentioned item. +const ORDERS: &str = r#" +name = "derived-differential-fixture" +[description] +en = "x" +de = "y" +[priority] +labels = ["P0", "P1", "P2", "P3", "P4"] +[types] +list = ["goal", "project", "action"] +[vocabulary.status] +open = "open" +in_progress = "in progress" +closed = "closed" +[ranking.next] +types = ["action", "project", "goal"] +order = [ { field = "priority", dir = "asc" } ] +[ranking.next.goal] +order = [ + { count = "contributors", type = "project", dir = "desc" }, + { field = "priority", dir = "asc" }, +] +[ranking.next.project] +order = [ + { related = "contributes_to", type = "goal", of = { field = "order" }, pick = "min", nulls = "last" }, + { related = "parent", type = "goal", of = "rank", pick = "max", nulls = "first" }, + { sum = [ { count = "mentioned_by" }, + { count = "mentioned_by", type = "action", label = "waiting", status = ["open", "in_progress"] } ], dir = "desc" }, + { count = "children", type = "action", dir = "desc" }, + { field = "priority", dir = "asc" }, +] +[ranking.next.action] +order = [ + { related = "parent", type = "project", of = "rank", nulls = "last" }, + { related = "dep", of = { field = "status", precedence = ["closed", "in_progress", "open"] }, pick = "min" }, + { count = "dependents", dir = "desc" }, + { related = "mentions", of = { field = "belongs_to" }, pick = "max", nulls = "last" }, + { count = "contributors", status = ["closed"], dir = "desc" }, + { field = "priority", dir = "asc" }, +] +[presentation.list] +columns = ["id"] +[fields.order] +type = "text" +on = ["goal"] +"#; + +inventory::submit!(PluginRegistration { + name: "derived-differential-fixture", + toml: ORDERS, + order: 140, +}); + +fn cfg() -> PluginConfig { + plugin::load("derived-differential-fixture").expect("the fixture is a valid plugin") +} + +/// The items of a generated board: three goals, five projects, eight actions. +fn items() -> Vec<(String, &'static str)> { + let mut out = Vec::new(); + for (ty, letter, n) in [("goal", 'g', 3), ("project", 'p', 5), ("action", 'a', 8)] { + for i in 0..n { + out.push((format!("ab12.{letter}{i:03}"), ty)); + } + } + out +} + +/// One replica's share of the writing. `added` holds the edges this replica added, so a remove +/// usually takes back a real one. +fn write(s: &mut Store, r: &mut Lcg, added: &mut Vec<(String, String, EdgeKind)>, steps: usize) { + let all = items(); + let pick = |r: &mut Lcg| all[r.next(all.len() as u64) as usize].0.clone(); + for step in 0..steps { + s.set_wall_clock(&format!( + "2026-10-0{}T{:02}:00:00Z", + 1 + r.next(7), + step % 24 + )); + let (x, y) = (pick(r), pick(r)); + match r.next(16) { + 0 => s.set_field(&x, "status", Some("in_progress".into()), "u"), + 1 | 2 => s.set_field(&x, "status", Some("closed".into()), "u"), + 3 => s.set_field(&x, "status", Some("open".into()), "u"), + 4 => s.set_field( + &x, + "archived", + Some(format!("2026-10-0{}", 1 + r.next(7))), + "u", + ), + 5 => s.set_field(&x, "archived", None, "u"), + 6 if r.next(3) == 0 => s.delete_item(&x, "u"), + 7 => s.set_field(&x, "priority", Some(r.next(5).to_string()), "u"), + 8..=12 => { + // Dependencies are rare: an active blocker takes its dependent out of `next`. + let kind = match r.next(10) { + 0..=2 => EdgeKind::Parent, + 3..=5 => EdgeKind::ContributesTo, + 6..=8 => EdgeKind::Mentions, + _ => EdgeKind::Dep, + }; + s.add_edge(&x, &y, kind, "u"); + added.push((x, y, kind)); + } + 13 if !added.is_empty() => { + let (from, to, kind) = added.swap_remove(r.next(added.len() as u64) as usize); + s.remove_edge(&from, &to, kind, "u"); + } + 14 => { + if r.next(3) == 0 { + s.remove_label(&x, "waiting", "u"); + } else { + s.add_label(&x, "waiting", "u"); + } + } + _ => s.set_custom_field_merge( + &x, + "order", + Some(r.next(6).to_string()), + "u", + MergeStrategy::Lww, + ), + } + } +} + +/// A merged board: two replicas that wrote apart, then exchanged everything. +fn gen_board(seed: u64) -> Store { + let mut r = Lcg(seed ^ 0xd1ff); + let mut a = Store::open_in_memory(1); + for (id, ty) in items() { + a.create_item(&id, ty, "T", "u"); + a.set_field(&id, "priority", Some(r.next(5).to_string()), "u"); + if ty == "goal" && r.next(4) != 0 { + a.set_custom_field_merge( + &id, + "order", + Some(r.next(6).to_string()), + "u", + MergeStrategy::Lww, + ); + } + } + let mut b = Store::open_in_memory(2); + b.apply(&a.export()); + let steps = 30 + r.next(40) as usize; + write(&mut a, &mut r, &mut Vec::new(), steps); + write(&mut b, &mut r, &mut Vec::new(), steps); + let from_b = b.export(); + a.apply(&from_b); + a +} + +/// `ops` folded into a fresh table. +fn fold(folder: &Folder, ops: &[(i64, nxs_foundation::model::Op)]) -> MemTable { + let t = MemTable::new("stream-1"); + ready(folder.fold_batch(&t, ops)).unwrap(); + t +} + +/// The served board with its ranking graph, as a server reads it for `reads` — and with the +/// contributes-to targets, so its tickets compare whole against `read::active_board`'s. +fn served(t: &MemTable, reads: &read::RankingReads) -> read::ActiveBoard { + let selection = ready(board::select(t)).unwrap(); + let plain = ready(records::active_board(t, selection)).unwrap(); + let plain = ready(records::with_contributes_to(t, plain)).unwrap(); + ready(records::with_ranking(t, plain, reads)).unwrap() +} + +fn ids(items: &[ItemRow]) -> Vec { + items.iter().map(|i| i.id.clone()).collect() +} + +fn count(s: &Store, sql: &str) -> usize { + s.connection() + .query_row(sql, [], |r| r.get::<_, i64>(0)) + .unwrap() as usize +} + +/// What the generated boards held, so the comparison is known to have compared something worth +/// comparing. +#[derive(Default)] +struct Seen(BTreeMap<&'static str, usize>); + +impl Seen { + fn add(&mut self, what: &'static str, n: usize) { + *self.0.entry(what).or_default() += n; + } + + fn assert_all(&self) { + for (what, n) in &self.0 { + assert!(*n > 0, "the generated boards have {what}: {:?}", self.0); + } + } +} + +#[test] +fn the_server_ranks_next_by_derived_keys_as_the_replica_does() { + let folder = Folder::platform().unwrap(); + let cfg = cfg(); + let reads = read::next_ranking_reads(&cfg) + .unwrap() + .expect("derived next keys"); + assert_eq!(reads.kinds.len(), 4, "every edge kind: {reads:?}"); + assert!(reads.labels && reads.custom, "{reads:?}"); + let mut seen = Seen::default(); + for seed in 0..60 { + let s = gen_board(seed); + let ctx = format!("seed {seed}"); + let t = fold(&folder, &delivered(&s, seed)); + + let local = read::with_ranking(&s, read::active_board(&s).unwrap(), &reads).unwrap(); + let server = served(&t, &reads); + assert_eq!(server.tickets, local.tickets, "active tickets, {ctx}"); + assert_eq!(server.ranking, local.ranking, "the ranking graph, {ctx}"); + + // 1 = 2 = 3: the SQL path, the Board path, the fold-ddb path. + let want = read::next(&cfg, &s, NOW, None).unwrap(); + assert_eq!( + ids(&read::next_active(&cfg, &local, NOW).unwrap()), + ids(&want), + "SQL path vs Board path, {ctx}" + ); + assert_eq!( + ids(&read::next_active(&cfg, &server, NOW).unwrap()), + ids(&want), + "SQL path vs fold-ddb path, {ctx}" + ); + assert_eq!( + read::next_active_value(&cfg, &server, NOW).unwrap(), + read::next_to_value_with_custom(&cfg, &s, &want).unwrap(), + "next JSON, {ctx}" + ); + + // Determinism: the same log folded in another order reads the same graph (every fourth + // board — a fold is most of a seed's time). + if seed % 4 == 0 { + let again = served(&fold(&folder, &delivered(&s, seed ^ 0xffff)), &reads); + assert_eq!(again.ranking, server.ranking, "another fold order, {ctx}"); + } + + // What the boards held. + let graph = server.ranking.as_ref().unwrap(); + let candidates: BTreeSet<&str> = want.iter().map(|i| i.id.as_str()).collect(); + seen.add("next rows", want.len()); + for n in graph.neighbours.values() { + let archived = n.item.archived.is_some(); + seen.add( + "closed neighbours", + usize::from(!archived && n.item.status.as_deref() == Some("closed")), + ); + seen.add("archived neighbours", usize::from(archived)); + seen.add("labelled neighbours", usize::from(!n.labels.is_empty())); + seen.add( + "goal neighbours with an order", + usize::from(n.custom.contains_key("order")), + ); + seen.add( + "neighbours with a parent", + usize::from(n.item.belongs_to.is_some()), + ); + } + for e in &graph.edges { + let into_candidate = candidates.contains(e.to.as_str()); + let from_neighbour = graph.neighbours.contains_key(&e.from); + seen.add( + "inbound contributes_to to a candidate", + usize::from(into_candidate && e.kind == EdgeKind::ContributesTo), + ); + seen.add( + "mentions of a candidate from an inactive item", + usize::from(into_candidate && from_neighbour && e.kind == EdgeKind::Mentions), + ); + seen.add( + "inactive children of a candidate", + usize::from(into_candidate && from_neighbour && e.kind == EdgeKind::Parent), + ); + seen.add( + "dep edges from a candidate to an inactive blocker", + usize::from( + candidates.contains(e.from.as_str()) + && graph.neighbours.contains_key(&e.to) + && e.kind == EdgeKind::Dep, + ), + ); + } + seen.add( + "edges dropped for a deleted end", + count( + &s, + "SELECT COUNT(*) FROM present_edges e JOIN items i + ON i.id IN (e.from_id, e.to_id) WHERE i.deleted = '1'", + ), + ); + // A child with two present parent edges: only `present_parent`'s pick may count. + seen.add( + "children parented twice", + count( + &s, + "SELECT COUNT(*) FROM (SELECT from_id FROM present_edges WHERE kind = 'parent' + GROUP BY from_id HAVING COUNT(*) > 1)", + ), + ); + seen.add( + "removed edges", + count(&s, "SELECT COUNT(*) FROM edge_removes"), + ); + // A derived key decides: a project ranked ahead of one with a better priority. + let projects: Vec<&ItemRow> = want + .iter() + .filter(|i| i.item_type.as_deref() == Some("project")) + .collect(); + seen.add( + "projects ordered against their priority", + projects + .windows(2) + .filter(|w| w[0].priority > w[1].priority) + .count(), + ); + } + seen.assert_all(); + for what in [ + "closed neighbours", + "archived neighbours", + "inbound contributes_to to a candidate", + "mentions of a candidate from an inactive item", + "inactive children of a candidate", + "dep edges from a candidate to an inactive blocker", + "edges dropped for a deleted end", + "children parented twice", + "projects ordered against their priority", + ] { + assert!(seen.0.contains_key(what), "{what} is counted"); + } +} + +#[test] +fn reading_the_ranking_graph_costs_one_query_per_ticket_and_per_live_neighbour() { + // §9 "bounded cost": every edge touching an active ticket is read once, from its entries; + // each inactive neighbour costs one GetItem and, when live, one Query for its parent plus one + // per join asked for — and nothing grows with history: an edge added and removed many times + // is still one entry per add, read in the same Query. + let folder = Folder::platform().unwrap(); + let cfg = cfg(); + let reads = read::next_ranking_reads(&cfg).unwrap().unwrap(); + let mut s = Store::open_in_memory(1); + s.create_item("ab12.g000", "goal", "Closed goal", "u"); + s.set_field("ab12.g000", "status", Some("closed".into()), "u"); + s.create_item("ab12.g001", "goal", "Deleted goal", "u"); + let n = 10u64; + for i in 0..n { + let id = format!("ab12.p{i:03}"); + s.create_item(&id, "project", "P", "u"); + for round in 0..5 { + s.add_edge(&id, "ab12.g000", EdgeKind::ContributesTo, "u"); + if round < 4 { + s.remove_edge(&id, "ab12.g000", EdgeKind::ContributesTo, "u"); + } + } + s.add_edge(&id, "ab12.g001", EdgeKind::Mentions, "u"); + } + s.delete_item("ab12.g001", "u"); + let t = fold(&folder, &delivered(&s, 1)); + let plain = ready(records::active_board(&t, ready(board::select(&t)).unwrap())).unwrap(); + let before = t.requests(); + let server = ready(records::with_ranking(&t, plain, &reads)).unwrap(); + let after = t.requests(); + // One adjacency Query per active ticket; the live neighbour's parent, labels and custom values. + assert_eq!(after.queries - before.queries, n + 3); + // One row per inactive neighbour: the closed goal and the deleted one. + assert_eq!(after.gets - before.gets, 2); + assert_eq!(after.index_queries, before.index_queries); + assert_eq!(after.writes, before.writes); + let local = read::with_ranking(&s, read::active_board(&s).unwrap(), &reads).unwrap(); + assert_eq!(server.ranking, local.ranking); + let graph = server.ranking.unwrap(); + assert_eq!( + graph.edges.len(), + n as usize, + "one contributes_to per project" + ); + assert_eq!( + graph.neighbours.keys().collect::>(), + ["ab12.g000"], + "the closed goal is a neighbour, the deleted one is not" + ); +} + +#[test] +fn a_stream_folded_by_index_revision_2_is_refused_then_refolded_and_ranks() { + // A stream folded before the adjacency entries said what their edge is (index revision 2): its + // entries hold only the tag, which would read no edge at all — so it is refused, and the fold + // run's rule refolds it. + let folder = Folder::platform().unwrap(); + let cfg = cfg(); + let reads = read::next_ranking_reads(&cfg).unwrap().unwrap(); + for seed in [3, 11, 27] { + let s = gen_board(seed); + let ops = delivered(&s, seed); + let t = fold(&folder, &ops); + for (sk, row) in t.rows() { + if let Some(Cell::Text(tag)) = row.get("tag").filter(|_| sk.starts_with(".adj#")) { + ready( + t.write(&Write { + sk: sk.clone(), + set: vec![("tag".into(), Cell::Text(tag.clone()))], + remove: ["from_id", "to_id", "kind", "lamport", "site", "removed"] + .map(String::from) + .to_vec(), + condition: Some(Cond::RowPresent), + }), + ) + .unwrap(); + } + } + ready(t.write(&Write { + sk: WATERMARK_KEY.into(), + set: vec![(format!("revision_{INDEX_DOMAIN}"), Cell::Int(2))], + remove: Vec::new(), + condition: Some(Cond::RowPresent), + })) + .unwrap(); + + let selection = ready(board::select(&t)).unwrap(); + match ready(records::active_board(&t, selection)) { + Err(records::RecordsError::NotCurrent { found: Some(2) }) => {} + other => panic!("seed {seed}: a revision-2 stream must be refused, got {other:?}"), + } + let mark = ready(Folder::watermark(&t)).unwrap().unwrap(); + assert!(!folder.is_current(&mark), "seed {seed}"); + ready(Folder::clear(&t)).unwrap(); + ready(folder.fold_batch(&t, &ops)).unwrap(); + + let server = served(&t, &reads); + assert_eq!( + ids(&read::next_active(&cfg, &server, NOW).unwrap()), + ids(&read::next(&cfg, &s, NOW, None).unwrap()), + "seed {seed}" + ); + } +} diff --git a/crates/fold-ddb/tests/dynamodb_local.rs b/crates/fold-ddb/tests/dynamodb_local.rs index 7c49649..d664758 100644 --- a/crates/fold-ddb/tests/dynamodb_local.rs +++ b/crates/fold-ddb/tests/dynamodb_local.rs @@ -1,6 +1,6 @@ //! The DynamoDB adapter against a real DynamoDB Local (6j6v.k7w7): the same fold written into //! DynamoDB and into a [`MemTable`] leaves the same items, the board reads back through the indexes -//! with the same lanes, and every request the client sends is recorded — none is a Scan. +//! with the same lanes and the same `typed` listings (6j6v.xkr7), and every request the client sends is recorded — none is a Scan. //! //! Gated like `nxs-server`'s suite: on the `dynamodb` feature and `NXF_RELAY_DDB_ENDPOINT`, skipped //! locally without one and a hard failure under CI. Each test creates its own table from @@ -16,7 +16,7 @@ use aws_sdk_dynamodb::config::{ use aws_sdk_dynamodb::error::BoxError; use aws_sdk_dynamodb::types::TableStatus; use aws_sdk_dynamodb::Client; -use common::{by_key, delivered, merged, NOW}; +use common::{by_key, delivered, merged, NOW, TYPES}; use nexus_flow_core::model::EdgeKind; use nexus_flow_core::store::Store; use nxs_fold_ddb::ddb::{create_table, DynamoDbTable}; @@ -204,6 +204,20 @@ async fn a_board_folded_into_dynamodb_reads_back_as_from_memory_and_no_request_i assert_eq!(got, want, "{lane:?} limit {limit:?}, seed {seed}"); } } + // The `typed` index (6j6v.xkr7): each type's live items, in id order, through the SDK. + for ty in TYPES { + let ids = |rows: Vec| -> Vec { + rows.iter() + .map(|r| text(r, "id").unwrap().to_string()) + .collect() + }; + let want = ids(mem.query_typed(ty).await.unwrap()); + let got = eventually(&want, || async move { + ids(ddb_ref.query_typed(ty).await.unwrap()) + }) + .await; + assert_eq!(got, want, "typed {ty}, seed {seed}"); + } // Two streams share the table without seeing each other: the second fold's partition // holds only its own rows (checked above), and its reads named its partition. let read = recorder.take(); @@ -309,7 +323,7 @@ async fn a_snapshot_taken_from_dynamodb_starts_another_stream_with_the_same_item // The partition and the index attributes name their stream; everything else is the // same across the two. r.remove("stream_id"); - for attr in ["nxf_active", "nxf_dated"] { + for attr in ["nxf_active", "nxf_dated", "nxf_typed"] { if let Some(nxs_foundation::change::Cell::Text(v)) = r.get_mut(attr) { *v = v .replacen("stream-a", "", 1) diff --git a/crates/fold-ddb/tests/fold_differential.rs b/crates/fold-ddb/tests/fold_differential.rs index ffb765f..824caa8 100644 --- a/crates/fold-ddb/tests/fold_differential.rs +++ b/crates/fold-ddb/tests/fold_differential.rs @@ -6,8 +6,8 @@ //! through the SQLite applier, in the order the replica took it in; and through this crate into a //! [`MemTable`], in a SHUFFLED order with ops delivered twice, as a relay without deduplication //! hands them over. Every row of every view table, every lane of the library and both dated lanes -//! must come out the same, and the incrementally kept indexes must be exactly what a full reindex -//! computes. +//! must come out the same, the `typed` index must list each type's live items (6j6v.xkr7), and the +//! incrementally kept indexes must be exactly what a full reindex computes. use nexus_flow_core::graph; use nexus_flow_core::model::EdgeKind; @@ -15,13 +15,13 @@ use nexus_flow_core::store::Store; use nexus_flow_core::task_reducer::TaskReducer; use nxs_fold_ddb::fold::Folder; use nxs_fold_ddb::mem::{ready, MemTable}; -use nxs_fold_ddb::table::{text, Dated}; +use nxs_fold_ddb::table::{text, Dated, Table}; use nxs_fold_ddb::{board, chunks}; use nxs_foundation::reducer::Reducer; use std::collections::BTreeMap; mod common; -use common::{merged, serve, served_rows, sqlite_rows, NOW}; +use common::{merged, serve, served_rows, sqlite_rows, NOW, TYPES}; fn dated_ids(t: &MemTable, lane: Dated) -> Vec { ready(board::dated(t, lane, None)) @@ -45,6 +45,7 @@ fn the_server_fold_in_any_order_answers_as_the_local_fold() { let folder = Folder::platform().unwrap(); let mut active_seen = 0; let mut chunks_seen = 0; + let mut typed_seen = 0; for seed in 0..120 { let s = merged(seed); let t = serve(&s, seed, &folder); @@ -95,6 +96,24 @@ fn the_server_fold_in_any_order_answers_as_the_local_fold() { "archived lane, {ctx}" ); + // The `typed` index (6j6v.xkr7): under each type exactly its live items, archived ones + // included, in id order. + for ty in TYPES { + let typed: Vec = ready(t.query_typed(ty)) + .unwrap() + .iter() + .map(|r| text(r, "id").unwrap().to_string()) + .collect(); + let want = sql_ids( + &s, + &format!( + "SELECT id FROM items WHERE COALESCE(deleted,'0')<>'1' AND type='{ty}' ORDER BY id" + ), + ); + typed_seen += want.len(); + assert_eq!(typed, want, "typed {ty}, {ctx}"); + } + // The flags kept op by op are exactly what recomputing them from the rows gives. let kept = t.rows(); ready(board::reindex(&t)).unwrap(); @@ -112,6 +131,7 @@ fn the_server_fold_in_any_order_answers_as_the_local_fold() { } assert!(active_seen > 0, "the generated boards have active tickets"); assert!(chunks_seen > 0, "the generated boards supersede chunks"); + assert!(typed_seen > 0, "the generated boards have typed items"); } #[test] diff --git a/crates/fold-ddb/tests/list_differential.rs b/crates/fold-ddb/tests/list_differential.rs new file mode 100644 index 0000000..c44e6fd --- /dev/null +++ b/crates/fold-ddb/tests/list_differential.rs @@ -0,0 +1,441 @@ +//! A server answers a ranked `list` as a replica does (6j6v.xkr7, docs/specs/derived-ranking-keys.md +//! §6 and §9 "differential"), and ranks a `next` whose keys read a `[ranking.list]` rank. +//! +//! Generated boards of four types — goals, projects, actions, persons — written by two replicas +//! apart and then merged: every status, archived and deleted items, retypes, priorities, the goals' +//! `order` custom field, `waiting` labels, and `parent`, `contributes_to`, `mentions` and `dep` +//! edges with removes. Each board is folded into a [`MemTable`] in a shuffled order with +//! duplicates, as the relay hands it over. +//! +//! The plugin ranks with every derived key kind across both sections: persons by a `sum` of +//! mentions, the `list` rank of the goals and the `next` rank of the projects they contribute to; +//! goals by their contributors under `[ranking.list]`; projects under `[ranking.next]` by the +//! `list` rank of their parent goal; actions under `[ranking.next]` by the `list` rank of the +//! persons they mention — the cross-section reads that were refused on the server until the +//! `typed` index — and, in `list`, by the `[ranking.list]` default order with a derived key. +//! +//! Three paths answer each of a set of `list` queries (types, none, container, label, status, +//! sort, limit): +//! +//! 1. the SQL path: `read::list_page` over the store; +//! 2. the Board path: `read::list_board_page` over the store's own [`read::ListBoard`] +//! (`read::list_board` for `read::list_reads`); +//! 3. the fold-ddb path: `read::list_board_page` over the folded tables' (`records::list_board`). +//! +//! All three must give the same rows, order and total, the two boards must be equal item for item +//! and edge for edge, and the JSON must be `Engine::list_query_value`'s. `next` is held to the +//! same three paths. + +use nexus_flow_core::model::{EdgeKind, ItemRow, MergeStrategy}; +use nexus_flow_core::store::Store; +use nexus_flow_facade::plugin::{self, PluginConfig, PluginRegistration}; +use nexus_flow_facade::read::{self, ListQuery, NextFilter, SortKey}; +use nxs_fold_ddb::fold::Folder; +use nxs_fold_ddb::mem::{ready, MemTable}; +use nxs_fold_ddb::{board, records}; +use std::collections::BTreeMap; + +mod common; +use common::{delivered, Lcg, NOW}; + +const ORDERS: &str = r#" +name = "list-differential-fixture" +[description] +en = "x" +de = "y" +[priority] +labels = ["P0", "P1", "P2", "P3", "P4"] +[types] +list = ["goal", "project", "action", "person"] +[vocabulary.status] +open = "open" +in_progress = "in progress" +closed = "closed" +[ranking.next] +types = ["action", "project"] +order = [ { field = "priority", dir = "asc" } ] +[ranking.next.project] +order = [ + { related = "parent", type = "goal", of = "rank", pick = "max", nulls = "first" }, + { related = "contributes_to", type = "goal", of = { field = "order" }, pick = "min", nulls = "last" }, + { count = "children", type = "action", dir = "desc" }, + { field = "priority", dir = "asc" }, +] +[ranking.next.action] +order = [ + { related = "mentions", type = "person", of = "rank", pick = "min", nulls = "last" }, + { related = "parent", type = "project", of = "rank", nulls = "last" }, + { related = "dep", of = { field = "status", precedence = ["closed", "in_progress", "open"] }, pick = "min" }, + { field = "priority", dir = "asc" }, +] +[ranking.list] +types = ["person", "goal", "project"] +order = [ + { count = "mentioned_by", dir = "desc" }, + { field = "priority", dir = "asc" }, +] +[ranking.list.goal] +order = [ + { count = "contributors", type = "project", dir = "desc" }, + { field = "priority", dir = "asc" }, +] +[ranking.list.person] +order = [ + { sum = [ { count = "mentioned_by", type = "project" }, + { count = "mentioned_by", type = "action", label = "waiting", status = ["open", "in_progress"] } ], dir = "desc" }, + { related = "contributes_to", type = "goal", of = "rank", pick = "min", nulls = "last" }, + { related = "contributes_to", type = "project", of = "rank", pick = "min", nulls = "last" }, + { count = "dependents", status = ["closed"], dir = "desc" }, + { field = "priority", dir = "asc" }, +] +[presentation.list] +columns = ["id"] +[fields.order] +type = "text" +on = ["goal"] +"#; + +inventory::submit!(PluginRegistration { + name: "list-differential-fixture", + toml: ORDERS, + order: 141, +}); + +fn cfg() -> PluginConfig { + plugin::load("list-differential-fixture").expect("the fixture is a valid plugin") +} + +const TYPES: [&str; 4] = ["goal", "project", "action", "person"]; + +/// The items of a generated board: three goals, four projects, six actions, five persons. +fn items() -> Vec<(String, &'static str)> { + let mut out = Vec::new(); + for (ty, letter, n) in [ + ("goal", 'g', 3), + ("project", 'p', 4), + ("action", 'a', 6), + ("person", 'u', 5), + ] { + for i in 0..n { + out.push((format!("ab12.{letter}{i:03}"), ty)); + } + } + out +} + +/// One replica's share of the writing. +fn write(s: &mut Store, r: &mut Lcg, added: &mut Vec<(String, String, EdgeKind)>, steps: usize) { + let all = items(); + let pick = |r: &mut Lcg| all[r.next(all.len() as u64) as usize].0.clone(); + for step in 0..steps { + s.set_wall_clock(&format!( + "2026-10-0{}T{:02}:00:00Z", + 1 + r.next(7), + step % 24 + )); + let (x, y) = (pick(r), pick(r)); + match r.next(18) { + 0 => s.set_field(&x, "status", Some("in_progress".into()), "u"), + 1 | 2 => s.set_field(&x, "status", Some("closed".into()), "u"), + 3 => s.set_field(&x, "status", Some("open".into()), "u"), + 4 => s.set_field( + &x, + "archived", + Some(format!("2026-10-0{}", 1 + r.next(7))), + "u", + ), + 5 => s.set_field(&x, "archived", None, "u"), + 6 if r.next(3) == 0 => s.delete_item(&x, "u"), + 7 => s.set_field(&x, "priority", Some(r.next(5).to_string()), "u"), + 8..=12 => { + let kind = match r.next(10) { + 0..=2 => EdgeKind::Parent, + 3..=5 => EdgeKind::ContributesTo, + 6..=8 => EdgeKind::Mentions, + _ => EdgeKind::Dep, + }; + s.add_edge(&x, &y, kind, "u"); + added.push((x, y, kind)); + } + 13 if !added.is_empty() => { + let (from, to, kind) = added.swap_remove(r.next(added.len() as u64) as usize); + s.remove_edge(&from, &to, kind, "u"); + } + 14 => { + if r.next(3) == 0 { + s.remove_label(&x, "waiting", "u"); + } else { + s.add_label(&x, "waiting", "u"); + } + } + 15 if r.next(3) == 0 => { + s.set_field(&x, "type", Some(TYPES[r.next(4) as usize].into()), "u") + } + _ => s.set_custom_field_merge( + &x, + "order", + Some(r.next(6).to_string()), + "u", + MergeStrategy::Lww, + ), + } + } +} + +/// A merged board: two replicas that wrote apart, then exchanged everything. +fn gen_board(seed: u64) -> Store { + let mut r = Lcg(seed ^ 0x115d); + let mut a = Store::open_in_memory(1); + for (id, ty) in items() { + a.create_item(&id, ty, "T", "u"); + a.set_field(&id, "priority", Some(r.next(5).to_string()), "u"); + if ty == "goal" && r.next(4) != 0 { + a.set_custom_field_merge( + &id, + "order", + Some(r.next(6).to_string()), + "u", + MergeStrategy::Lww, + ); + } + } + let mut b = Store::open_in_memory(2); + b.apply(&a.export()); + let steps = 30 + r.next(40) as usize; + write(&mut a, &mut r, &mut Vec::new(), steps); + write(&mut b, &mut r, &mut Vec::new(), steps); + let from_b = b.export(); + a.apply(&from_b); + a +} + +fn ids(items: &[ItemRow]) -> Vec { + items.iter().map(|i| i.id.clone()).collect() +} + +/// The `list` queries each board is asked: a type, several, none; a container, a label, a status; +/// the default order; a cut. +fn queries(container: &str) -> Vec { + let ranked = || ListQuery::new().sort(SortKey::Rank).now(NOW); + let types = |tys: &[&str]| tys.iter().fold(NextFilter::new(), |f, t| f.with_type(*t)); + vec![ + ranked().filter(types(&["person"])), + ranked().filter(types(&["goal"])), + ranked().filter(types(&["project", "person"])), + ranked(), + ranked().filter(types(&["action"]).with_container(container)), + ranked().filter(NextFilter::new().with_label("waiting")), + ranked().status("closed").filter(types(&["person"])), + ListQuery::new().filter(types(&["person"])), + ranked().filter(types(&["person"])).limit(3), + ] +} + +/// What the generated boards held, so the comparison is known to have compared something worth +/// comparing. +#[derive(Default)] +struct Seen(BTreeMap<&'static str, usize>); + +impl Seen { + fn add(&mut self, what: &'static str, n: usize) { + *self.0.entry(what).or_default() += n; + } +} + +#[test] +fn the_server_answers_a_ranked_list_as_the_replica_does() { + let folder = Folder::platform().unwrap(); + let cfg = cfg(); + let mut seen = Seen::default(); + for seed in 0..40 { + let s = gen_board(seed); + let ctx = format!("seed {seed}"); + let t = MemTable::new("stream-1"); + ready(folder.fold_batch(&t, &delivered(&s, seed))).unwrap(); + + for (n, q) in queries(&format!("ab12.p{:03}", seed % 4)) + .iter() + .enumerate() + { + let ctx = format!("{ctx}, query {n}"); + let reads = read::list_reads(&cfg, q); + let local = read::list_board(&s, &reads).unwrap(); + let server = ready(records::list_board(&t, &reads)).unwrap(); + assert_eq!(server.items, local.items, "listed items, {ctx}"); + assert_eq!(server.ranking, local.ranking, "the ranking graph, {ctx}"); + assert_eq!( + server.active.as_ref().map(|a| &a.tickets), + local.active.as_ref().map(|a| &a.tickets), + "next's board, {ctx}" + ); + + // 1 = 2 = 3: the SQL path, the Board path, the fold-ddb path. + let want = read::list_page(&cfg, &s, q).unwrap(); + let board_path = read::list_board_page(&cfg, &local, q).unwrap(); + let served = read::list_board_page(&cfg, &server, q).unwrap(); + assert_eq!( + ids(&board_path.items), + ids(&want.items), + "SQL vs Board, {ctx}" + ); + assert_eq!( + ids(&served.items), + ids(&want.items), + "SQL vs fold-ddb, {ctx}" + ); + assert_eq!(served.total, want.total, "total, {ctx}"); + let records = read::list_to_value_with_custom(&cfg, &s, &want.items).unwrap(); + assert_eq!( + read::list_board_value(&cfg, &server, q).unwrap(), + read::next_page_envelope(records, &want, false), + "list JSON, {ctx}" + ); + + seen.add("listed rows", want.items.len()); + if let Some(graph) = &server.ranking { + seen.add("graph listed items", graph.listed.len()); + seen.add("graph edges", graph.edges.len()); + } + seen.add( + "boards with next's board", + usize::from(server.active.is_some()), + ); + seen.add( + "listed items with a closed-parent reason", + server + .items + .values() + .filter(|i| !i.parent_closed.is_empty()) + .count(), + ); + seen.add( + "closed persons listed", + want.items + .iter() + .filter(|i| { + i.item_type.as_deref() == Some("person") + && i.status.as_deref() == Some("closed") + }) + .count(), + ); + } + + // `next` whose keys read the `list` rank of persons and goals (spec §12 point 1). + let reads = read::next_ranking_reads(&cfg) + .unwrap() + .expect("derived next keys"); + let selection = ready(board::select(&t)).unwrap(); + let plain = ready(records::active_board(&t, selection)).unwrap(); + let plain = ready(records::with_contributes_to(&t, plain)).unwrap(); + let server = ready(records::with_ranking(&t, plain, &reads)).unwrap(); + let local = read::with_ranking(&s, read::active_board(&s).unwrap(), &reads).unwrap(); + assert_eq!(server.ranking, local.ranking, "next's graph, {ctx}"); + let want = read::next(&cfg, &s, NOW, None).unwrap(); + assert_eq!( + ids(&read::next_active(&cfg, &local, NOW).unwrap()), + ids(&want), + "next: SQL vs Board, {ctx}" + ); + assert_eq!( + ids(&read::next_active(&cfg, &server, NOW).unwrap()), + ids(&want), + "next: SQL vs fold-ddb, {ctx}" + ); + seen.add("next rows", want.len()); + seen.add( + "archived items", + s.list_items() + .unwrap() + .iter() + .filter(|i| i.archived.is_some() && i.deleted.as_deref() != Some("1")) + .count(), + ); + } + for what in [ + "listed rows", + "graph listed items", + "graph edges", + "boards with next's board", + "listed items with a closed-parent reason", + "closed persons listed", + "next rows", + "archived items", + ] { + assert!( + seen.0.get(what).copied().unwrap_or(0) > 0, + "the generated boards have {what}: {:?}", + seen.0 + ); + } +} + +#[test] +fn a_stream_not_folded_by_this_index_revision_is_refused_for_a_list() { + // The `typed` index and the self-describing adjacency entries come with index revision 3: a + // stream folded before has neither, and its list would come back empty — refused instead. + let folder = Folder::platform().unwrap(); + let cfg = cfg(); + let s = gen_board(5); + let t = MemTable::new("stream-1"); + ready(folder.fold_batch(&t, &delivered(&s, 5))).unwrap(); + use nxs_fold_ddb::fold::INDEX_DOMAIN; + use nxs_fold_ddb::layout::WATERMARK_KEY; + use nxs_fold_ddb::table::Table; + use nxs_fold_ddb::write::{Cond, Write}; + use nxs_foundation::change::Cell; + ready(t.write(&Write { + sk: WATERMARK_KEY.into(), + set: vec![(format!("revision_{INDEX_DOMAIN}"), Cell::Int(2))], + remove: Vec::new(), + condition: Some(Cond::RowPresent), + })) + .unwrap(); + let q = &queries("ab12.p000")[0]; + match ready(records::list_board(&t, &read::list_reads(&cfg, q))) { + Err(records::RecordsError::NotCurrent { found: Some(2) }) => {} + other => panic!("a revision-2 stream must be refused, got {other:?}"), + } +} + +#[test] +fn reading_a_typed_list_costs_one_index_query_and_four_requests_per_item() { + // §9 "bounded cost": the persons of a board are one Query on the `typed` index, and each costs + // its adjacency, labels, custom values and timestamps — whatever else the board holds, which + // this list reads nothing of. A type filter reads only its own type. + let folder = Folder::platform().unwrap(); + let cfg = cfg(); + let mut s = Store::open_in_memory(1); + let n = 8u64; + for i in 0..n { + s.create_item(&format!("ab12.u{i:03}"), "person", "P", "u"); + } + for i in 0..30 { + s.create_item(&format!("ab12.a{i:03}"), "action", "A", "u"); + } + s.create_item("ab12.uzzz", "person", "Archived", "u"); + s.set_field("ab12.uzzz", "archived", Some("2026-10-01".into()), "u"); + let t = MemTable::new("stream-1"); + ready(folder.fold_batch(&t, &delivered(&s, 1))).unwrap(); + let q = ListQuery::new().filter(NextFilter::new().with_type("person")); + let reads = read::list_reads(&cfg, &q); + assert_eq!(reads.ranking, None); + assert!(!reads.next); + let before = t.requests(); + let server = ready(records::list_board(&t, &reads)).unwrap(); + let after = t.requests(); + assert_eq!(after.index_queries - before.index_queries, 1); + assert_eq!(after.queries - before.queries, 3 * n); + // The watermark, and each person's timestamps; no parents. + assert_eq!(after.gets - before.gets, 1 + n); + assert_eq!(after.writes, before.writes); + assert_eq!( + server.items.len(), + n as usize, + "the archived person is not listed" + ); + let page = read::list_board_page(&cfg, &server, &q).unwrap(); + assert_eq!( + ids(&page.items), + ids(&read::list_page(&cfg, &s, &q).unwrap().items) + ); +} diff --git a/crates/fold-ddb/tests/robustness.rs b/crates/fold-ddb/tests/robustness.rs index 414a3f3..815e3b1 100644 --- a/crates/fold-ddb/tests/robustness.rs +++ b/crates/fold-ddb/tests/robustness.rs @@ -1,6 +1,7 @@ //! What a server fold does when things go wrong (review of PR #24): a run that dies at any write //! and folds its batch again, an op the table cannot hold, a clear that dies half way, a release -//! with other fold revisions, a lane cut by a limit, and a snapshot that does not fit. +//! with other fold revisions, a lane cut by a limit, a snapshot that does not fit — and the `typed` +//! index kept through retypes, archives and deletes (6j6v.xkr7). mod common; @@ -94,6 +95,10 @@ impl Table for Failing<'_> { async fn query_dated(&self, lane: Dated, limit: Option) -> Result, Injected> { Ok(ok(self.inner.query_dated(lane, limit).await)) } + + async fn query_typed(&self, item_type: &str) -> Result, Injected> { + Ok(ok(self.inner.query_typed(item_type).await)) + } } /// Fold `first` cleanly, then `second` with a run that dies at each write in turn and folds @@ -214,6 +219,107 @@ fn an_op_the_table_cannot_hold_is_refused_and_the_stream_folds_on() { assert_eq!(lanes.ready, vec!["ab12.0001", "ab12.0002"]); } +#[test] +fn a_type_too_long_for_the_typed_index_is_refused_and_the_item_folds_on_without_it() { + // 6j6v.xkr7: the `typed` index partition carries the type; DynamoDB refuses an index key past + // 2,048 bytes, and a refused write would fail every run at the same op. So the op that sets + // such a type is refused before it writes, as an op with a key too long is, and the item keeps + // the type it had. + let folder = Folder::platform().unwrap(); + let mut s = Store::open_in_memory(1); + s.create_item("ab12.0001", "task", "T", "u"); + s.set_field("ab12.0001", "type", Some("t".repeat(2100)), "u"); + s.create_item("ab12.0002", &"n".repeat(2030), "Just fits", "u"); + let t = MemTable::new("stream-1"); + let refused = ready(folder.fold_batch(&t, &numbered(&s, 0))).unwrap(); + let reasons: Vec<&str> = refused.iter().map(|r| r.reason.as_str()).collect(); + assert_eq!(refused.len(), 1, "{reasons:#?}"); + assert!( + reasons[0].contains("`typed` index partition"), + "{reasons:#?}" + ); + let ids = |ty: &str| -> Vec { + ready(t.query_typed(ty)) + .unwrap() + .iter() + .map(|r| text(r, "id").unwrap().to_string()) + .collect() + }; + assert_eq!(ids("task"), ["ab12.0001"]); + assert_eq!(ids(&"n".repeat(2030)), ["ab12.0002"]); +} + +#[test] +fn retyping_archiving_and_deleting_move_an_item_on_the_typed_index() { + // 6j6v.xkr7: an item is listed under its type while it is live — archived or not — moves with a + // retype, and leaves when it is deleted. A type with `#` in it gets its own partition. + let folder = Folder::platform().unwrap(); + let mut s = Store::open_in_memory(1); + s.create_item("ab12.0001", "person", "Ada", "u"); + s.create_item("ab12.0002", "person", "Bob", "u"); + s.create_item("ab12.0003", "a#b", "Odd", "u"); + let t = MemTable::new("stream-1"); + let mut seen = 0; + let mut step = |s: &Store| { + let ops = numbered(s, seen); + seen += ops.len(); + ready(folder.fold_batch(&t, &ops)).unwrap(); + ["person", "project", "a#b", "a"].map(|ty| { + ready(t.query_typed(ty)) + .unwrap() + .iter() + .map(|r| text(r, "id").unwrap().to_string()) + .collect::>() + }) + }; + let v = |ids: &[&str]| ids.iter().map(|s| s.to_string()).collect::>(); + let none = v(&[]); + + assert_eq!( + step(&s), + [ + v(&["ab12.0001", "ab12.0002"]), + none.clone(), + v(&["ab12.0003"]), + none.clone() + ] + ); + s.set_field("ab12.0002", "archived", Some("2026-10-08".into()), "u"); + s.set_field("ab12.0001", "status", Some("closed".into()), "u"); + assert_eq!( + step(&s), + [ + v(&["ab12.0001", "ab12.0002"]), + none.clone(), + v(&["ab12.0003"]), + none.clone() + ], + "closed and archived items stay listed" + ); + s.set_field("ab12.0001", "type", Some("project".into()), "u"); + assert_eq!( + step(&s), + [ + v(&["ab12.0002"]), + v(&["ab12.0001"]), + v(&["ab12.0003"]), + none.clone() + ] + ); + s.delete_item("ab12.0002", "u"); + s.delete_item("ab12.0003", "u"); + assert_eq!( + step(&s), + [none.clone(), v(&["ab12.0001"]), none.clone(), none.clone()] + ); + // The attribute is on the item rows only, never on an adjacency entry or another row. + assert!(t + .rows() + .iter() + .filter(|(_, r)| r.contains_key("nxf_typed")) + .all(|(sk, _)| sk.starts_with("items#"))); +} + #[test] fn a_clear_that_dies_keeps_the_watermark_and_a_release_with_other_rules_refolds_from_zero() { let folder = Folder::platform().unwrap(); diff --git a/docs/specs/E4-sync-protocol-server.md b/docs/specs/E4-sync-protocol-server.md index a63c679..9dc3d5c 100644 --- a/docs/specs/E4-sync-protocol-server.md +++ b/docs/specs/E4-sync-protocol-server.md @@ -278,6 +278,13 @@ relay — does it with `nxs-fold-ddb`: the reducers a local replica registers, t - **GSI `dated`** (`nxf_dated` / `nxf_dated_at`): sparse — `#closed` by `closed_at`, `#archived` by `archived`, newest first; the sort key is `U+001F`, so tickets of one instant keep one order and a `limit` cuts the lane at the same place every time. +- **GSI `typed`** (`nxf_typed` / `sk`, both S, projecting all; added 2026-10-11, `6j6v.xkr7`): + sparse — `#` (the type escaped as a key value: `%` → `%25`, `#` → `%23`) on every + live item row, closed and archived ones included, removed on a delete. One Query lists a type's + live items, the `list` scope a ranked hosted `list` and the derived ranking keys rank over + (`records::list_board`, `docs/specs/derived-ranking-keys.md` §6). The deploying stack adds the + GSI first: a fold against a table without it still writes the attribute harmlessly, and only + the read needs the index. - **The active tickets' records** (added 2026-10-10, `6j6v.t1ym`). The LANES need only the `active` Query above. The RECORDS a lane is shown with need more: a lane's JSON carries joins kept in other tables — labels, custom values, timestamps, bearing thread links, the parent's diff --git a/docs/specs/derived-ranking-keys.md b/docs/specs/derived-ranking-keys.md index 22c2d7f..3682d4f 100644 --- a/docs/specs/derived-ranking-keys.md +++ b/docs/specs/derived-ranking-keys.md @@ -2,7 +2,10 @@ **Status: decided by the owner on 2026-10-09** (§10). The cuts that build it are listed in §11. **Built locally (x0wf, §12):** the three key kinds, their load-time rules and the type chain over -the SQL path, in `next` and `list`. The server cuts (1c5s, xkr7) are open. +the SQL path, in `next` and `list`. **Built on the server for `next` (1c5s, §6):** the same keys +over the hosted board, with the rule the owner decided on 2026-10-11 (§10.5). **Built on the server +for `list` (xkr7, §6):** the `typed` index, a ranked hosted `list`, and a hosted `next` whose keys +read a `[ranking.list]` rank. Fourth cut of `6j6v.kfs7` (cut with the owner on 2026-10-09). The three cuts before it add the plumbing this one ranks with: @@ -135,11 +138,45 @@ derived key pays nothing. `nxs-fold-ddb` answers `next` from a sparse index of ACTIVE tickets through `graph::Board` (6j6v.k7w7). Derived keys must give the same order there. -**`next`.** -- `Board` gains the edge kinds the keys need (`contributes_to`, `mentions`), and each ticket's - type, status, labels and the fields the keys read. -- The rule that makes this possible is the one `Board` already has: a counterpart that is not an - active ticket counts as absent, which is exactly §4's scope for `next`. +**`next`** (built in 1c5s; decided by the owner on 2026-10-11, §10.5). +- **A key reads every LIVE neighbour, on the server as locally** (§12, "Live neighbours"): a + deleted neighbour is gone, a closed or archived one counts. This replaces the first reading of + this section, that a counterpart which is not an active ticket counts as absent. That rule stays + the rule of the LANES (`graph::Board`); it is not the rule of the keys. Only the scope of + `of = "rank"` is the active set: a neighbour outside `next`'s candidates has no rank (§4), on + both paths. +- **Where it lives.** The facade's `ActiveBoard` — the store-free lanes' input, `graph::Board` + plus each active ticket's row and joins — gains a `RankingGraph`: every present edge of the kinds + the keys name (`contributes_to`, `mentions`, `dep`, `parent` as the `present_parent` projection) + with an active end and both ends live, and each live neighbour that is not active with its row, + labels and custom values. `graph::Board` itself is unchanged: it holds no rows, and the lanes read + no neighbour that is not active. `read::next_ranking_reads(cfg)` names what the graph must hold + for a plugin; `read::next_active` refuses a plugin with derived keys while its board lacks it, and + ranks by the same evaluation the store path runs once it is there. +- **What the fold writes.** Every edge add already had an adjacency entry under each end + (`.adj##`), whatever its kind and its ends' activity. Since index revision 3 each entry + also copies the add's `from_id`, `to_id`, `kind`, `lamport` and `site` and says whether the add is + removed, kept up to date when the remove is folded. So one Query on an item's prefix reads every + edge touching it, in and out, with its presence: inbound `contributes_to`, `mentions` both ways, + and the `parent` edges a child's current parent is picked from. No new index, no new key + attribute: the TABLE CONTRACT IS UNCHANGED. Revision 3 makes every stream folded before it not + current; it answers `NotCurrent` until the fold run has folded it again. +- **What the server reads** (`records::with_ranking`): one Query per active ticket on its + adjacency; per neighbour that is not active one `GetItem` of its row and, when it is live, one + Query on its own adjacency for its current parent, plus one Query each for its labels and custom + values when a key reads them. +- **A `[ranking.list]` rank in a `next` key** (refused on the server in 1c5s, ranked since xkr7): a + `[ranking.next]` order that reads — directly or down its type chain — the rank of a type only + `[ranking.list]` orders (§12, first point). That rank is computed over the type's live, + unarchived items, which the active board does not hold; `read::next_ranking_reads` names such a + type in `RankingReads::list_types`, and the server reads its items off the `typed` index into the + graph (`RankingGraph::listed`), with the edges around them. +- **What xkr7 builds on.** The self-describing adjacency entries are written for EVERY edge between + ANY items, so xkr7's "widened adjacency" needed no new entries for the edges: a `list`-ranked item + reads its edges with the same Query (verified in xkr7: its `.adj#` entries give its current + parent, its `contributes_to` targets, its parents for `parent_closed_reason` and every edge a key + follows or counts). What xkr7 adds is the `typed` index (the `list` scope of a type), the read + that ranks a type's live items from it, and lifting the refusal above. **`list` needs a second index (decided §10.4).** Objects that `list` ranks, such as a person, are long-lived and have no "active" status. If they did, they would rank through `next`, and `list` @@ -155,10 +192,11 @@ gains a second sparse index, keyed by type: - **Generic.** The fold indexes every live item by its type, so it needs no plugin knowledge of which types `list` ranks. The price is one index write per item, not only for the ranked types. - **Edges.** A derived key on a `list`-ranked type reads that item's `contributes_to` and `mentions` - neighbours, and the counts read their inbound edges. Today the adjacency entries (`.adj#`) are - kept only for `dep`/`parent` edges that touch an active ticket. They are widened to every present - edge of a kind some key names, between live items. That widening is the open design work of the - server cut (§11), together with what it costs per edge write. + neighbours, and the counts read their inbound edges. Corrected in 1c5s: the adjacency entries + (`.adj#`) were always written for every edge add, under both ends; only the `active` INDEX flag is + limited to `dep`/`parent` edges with an active end. Since index revision 3 the entries say what + their edge is and whether it is removed (`next` above), so a `list`-ranked item reads its edges + with one Query on its own prefix, at no extra cost per edge write. - **Archived items.** They are in the index, because they are live. `list` filters them at read, as it does locally. - **It changes the table contract.** The deploying stack (the web package's CDK) must add the GSI @@ -166,6 +204,32 @@ gains a second sparse index, keyed by type: still writes the attribute, which costs nothing. Only the `list` read needs the index. The rollout order is therefore: the stack first, then the read. +**`list` as built (xkr7).** +- **The key.** `nxf_typed` is `#` with the type escaped as a key value is (`%` → + `%25`, `#` → `%23`), so `a` + `b#c` and `a#b` + `c` cannot share a partition; the sort key is the + table's `sk`, so a Query lists a type's items in id order. An item without a type is on no + partition. A type whose partition would pass the 2,048 bytes DynamoDB allows an index key is + refused at fold time, as an op with a sort key too long is. +- **Index revision.** The attribute ships in revision 3 with the self-describing adjacency entries + (1c5s and xkr7 reach the same beta), so revision 3 still makes every older stream not current + and one refold sets both; no revision 4. +- **Where it lives.** The facade's `read::list_reads(cfg, query)` names what a hosted `list` reads + (`ListReads`: the listed types — the query's type filter, else every type —, whether `next`'s + board is needed, and the graph); `nxs_fold_ddb::records::list_board` reads it into a `ListBoard` + (the listed items with the joins a `list` record and its filters carry, `next`'s board, the + `RankingGraph`); `read::list_board_page` and `read::list_board_value` filter, rank and project it + by the code `list_page` runs, and `read::list_board` reads the same board from a store, the + parity oracle. +- **Without a type filter** the server lists every live item: it reads the partition's `items#` + rows (strongly consistent, one Query, no index), since the index lists one type at a time. +- **Paging.** As for the hosted `next`: the store path's page cache is machine-local and a + server's paging is an open decision (15ed), so the store-free `list` is not paged — `limit` cuts + it with the total disclosed, `paginate` and a token are refused with `validation`, and a server + cuts the list itself. +- **`now`.** A `list` key that reads a rank `next` gives takes `next`'s candidates at the query's + `now`, as locally (§12); without one the read answers `validation` where a ranked row needs it, + and `next`'s board is not read for it. + A differential test holds the SQL path and the server path to the same order on generated boards, for `next` and for `list`, as for the lanes today. @@ -222,8 +286,13 @@ the plugin declares (`plugin-custom-fields.md`), as decided in §10.2. - **Per kind:** unit tests over hand-built boards: a value from one neighbour, from several (`min`/`max`), from none (null placement); counts with each filter. - **Chains:** actions by project rank by goal order across three types, and a refused cycle. -- **Differential:** SQL path against `graph::Board` on generated boards with derived keys, as - `fold_differential` does for the lanes. +- **Differential:** SQL path against the server path on generated boards with derived keys, as + `fold_differential` does for the lanes. Built for `next` in 1c5s + (`crates/fold-ddb/tests/derived_differential.rs`): the SQL path, the store-free path over the + store's own board and graph, and the same over the folded tables give one order. Built for + `list` in xkr7 (`crates/fold-ddb/tests/list_differential.rs`): the same three paths on + generated boards of four types with retypes, every key kind and cross-section ranks both ways, + over a set of queries (types, none, container, label, status, sort, limit), and the JSON. - **Determinism:** shuffled edge order gives the same result, and ties fall to the id. - **Bounded cost:** a ratio guard in the style of `xn8s`. A query with derived keys runs the edge pass once, not once per candidate. @@ -245,15 +314,29 @@ the plugin declares (`plugin-custom-fields.md`), as decided in §10.2. items was declined: the objects `list` ranks have no active status by design. If they had one, they would rank through `next`, and `list` would not be needed. Leaving `list` unranked on the server was declined because the product reads through the hosted path. +5. **On the server a key reads every live neighbour** (owner, 2026-10-11, during 1c5s). The first + text of §6 said a counterpart that is not an active ticket counts as absent; the local build + (§12) reads every live neighbour, closed and archived ones included. The owner chose the local + rule for both paths, so a server and a replica rank alike: the fold keeps self-describing + adjacency entries (index revision 3) and the server reads the neighbours that are not active. + Narrowing the local rule to active neighbours was declined, as was refusing on the server every + key whose value can depend on a neighbour that is not active (it would refuse §8's first order). + A `[ranking.next]` order that reads the rank of a type only `[ranking.list]` orders was refused on + the server until the `typed` index (xkr7) gave it that type's scope; it ranks there since. ## 11. Cuts - **Derived keys, local.** The three key kinds, load-time validation and the type chain over the SQL path. Usable in `next` (after `z9jk`) and `list` (after `n698`). -- **Derived keys on the server, `next`.** `graph::Board` with the extra edges and fields; the - differential test. -- **The `typed` index and a ranked hosted `list`.** The GSI, the widened adjacency, the read, and - the table-contract note for the deploying stack, coordinated with app-foundations. +- **Derived keys on the server, `next`** (1c5s, built). The `RankingGraph` on the store-free + lanes' board, the self-describing adjacency entries (index revision 3), and the differential test + (SQL path = Board path = fold-ddb path on generated boards with closed, archived and deleted + neighbours, mentions and inbound `contributes_to`). +- **The `typed` index and a ranked hosted `list`** (xkr7, built). The GSI, the read + (`ListBoard`, `records::list_board`, `read::list_board_page`), the `[ranking.list]` rank in a + hosted `next`, the differential test, and the table-contract note for the deploying stack + (`nxs-fold-ddb` crate doc, E4 §5.1), coordinated with app-foundations. The widened adjacency + turned out to be 1c5s's self-describing entries; nothing more was needed. - **Shared with `6j6v.q29p`.** The reverse mention lookup and `count = "mentioned_by"` use one implementation. @@ -286,7 +369,10 @@ reading of the text, so a later decision can loosen it without breaking a declar `blocked` and the lanes' `--sort rank`, which compute none. The `[ranking.list]` default order takes them. - **`nulls` on a `count` or `sum`** is refused: a count is never null (§3.2). -- **The store-free lanes** (`read::next_active`) refuse a `[ranking.next]` with derived keys until - the server cut (1c5s) gives them the edges. +- **The store-free lanes** (`read::next_active`) rank a `[ranking.next]` with derived keys from + the board's `RankingGraph` since 1c5s (§6), with the live-neighbour rule above; since xkr7 an + order that reads the rank of a type only `[ranking.list]` orders too, from the graph's listed + items. The store-free `list` (`read::list_board_page`, xkr7) ranks a `[ranking.list]` the same + way. - **TOML.** An inline table may not span lines outside its arrays (TOML 1.0), so a `sum`'s `dir` stands on the line its array closes on. §8 above breaks that line for reading only.